metaflow

Why we included this project

Metaflow is a Python framework for teams that want a single tool to carry an ML project from a notebook experiment to a scheduled production run. You describe the pipeline as a flow in code, and the framework takes care of versioning, artifact storage, and scaling, whether you are running on a laptop or on cloud and Kubernetes infrastructure. The same flow you prototype in a notebook can be promoted to production without being rewritten, which removes one of the most annoying transitions in real ML work. It integrates with AWS, Azure, and GCP and with Argo Workflows or Airflow, so you do not have to adopt a new infrastructure stack. Metaflow comes out of Netflix, where it supports thousands of ML projects, and it suits data scientists and ML engineers who need their experimentation workflow to survive contact with production.