I would also add - dependency management (certain degree of reproducibility) as a first class feature leveraging conda.

So it's like Snakemake? [0] Snakemake can also control k8s clusters. Perhaps they are easier to set up with Metaflow?

[0] https://snakemake.readthedocs.io/en/stable/

Addendum - You can mix and match what steps of the DAG run on the cloud.

looks like airflow with ML tools?

It provides a Python DAG building library like Airflow, but doesn't do Airflow's 'Operator ecosystem' thing.

It also is very opinionated about dependency management (Conda-only) and is Python-only, where Airflow I think has operators to run arbitrary containers. So Metaflow is a non-starter I think if you don't want to exclusively use Python.

Airflow also ships with built-in scheduler support (Celery?) or can run on K8s. Metaflow doesn't have this. Seems to rely on AWS Batch for production DAG execution.

Airflow ships with a pretty rich UI. Metaflow seems to be anti-UI, and provides a novel Notebook-oriented workflow interaction model.

Metaflow has pretty nice code artifact + params snapshotting functionality which is a core selling point. Airflow is not as supportive of this so it's harder to do reproducibility (I think). This is encapsulated by their "Datastore" model which can locally or in S3 persist flow code, config and data.

Metaflow does come bundled with a scheduler that can place jobs on a variety of compute platforms (current release supports local on-instance and AWS batch). In terms of dependencies, we went with conda because of its traction in the data science community as well as excellent support for system packages. Our execution model also supports arbitrary docker containers (on AWS batch) where you can theoretically bake in your own dependencies. In terms of language support, we have bindings for R internally, that we plan to open source as well.

I wouldn’t qualify metaflow as anti-UI. For model monitoring, we haven’t found a good enough UI that can handle the diversity of models and use cases we see internally, and believe that notebooks are an excellent visualisation medium that gives the power to the end user (data scientists) to craft dashboards as they see fit. For tracking the execution of production runs, we have historically relied on the UI of the scheduler itself (meson). We are exploring what a metaflow-specific UI might look like.

As for comparisons with Airflow, it is an excellent production grade scheduler. Metaflow intends to solve a different problem of providing an excellent development and deployment experience for ML pipelines.

Thanks for these clarifications.

> Our execution model also supports arbitrary docker containers (on AWS batch) where you can theoretically bake in your own dependencies.

That's fair, but it doesn't seem to be something encouraged by the framework, and that's fine.

> I wouldn’t qualify metaflow as anti-UI.

Maybe anti-UI is too strong yeah. I personally think your approach could be great. Looking forward to exploring it.

Thanks for open sourcing this! This seems like precisely the kind of tool I‘ve been looking for. One thing that would be really great is support for HPC schedulers like SLURM. It seems relatively straightforward to add, so I might give it a shot myself.

Happy to help either through our gitter chat or help@metaflow.org.

> Metaflow has pretty nice code artefact + params snapshotting functionality which is a core selling point.

How would you say this aspect Metaflow compares to Git LFS (https://git-lfs.github.com) and Data Version Control (https://dvc.org)?

Also, thank you to the Netflix team for creating such a wonderful library and for making it open source.

I guess (?) - minus the input spec being not YAML but more language native (pythonic for e.g.)