hamilton

Why we included this project

Most data teams end up with the same problem: feature pipelines and ETL jobs that start as a few scripts and notebooks, then grow into something nobody wants to rerun or refactor. Hamilton sidesteps that by treating each transformation as a plain Python function and letting the framework infer the DAG from function parameters, so you get structure without a heavyweight framework forcing itself onto your code. Because execution is decoupled from the graph definition, the same code can run locally for quick iteration, then on Spark or DuckDB when the data outgrows your laptop. Every step also records lineage and metadata, which makes tracing where a value came from and auditing a data flow far less painful than digging through logs. If you want modular, self-documenting pipelines without adopting an orchestrator, this is a practical middle ground.