Metaflow comes with a built in scheduler. If your company has an existing scheduler, e.g. for ETL, you can translate Metaflow DAGs to the production scheduler automatically. This is what we do at Netflix.
We could provide similar support e.g. for Airflow, if there's interest.
For monitoring, we have relied on notebooks this far. Their flexibility and customizability is unbeatable. We might build some lightweight discovery / debugging UI later, if there's demand. We are a bit on the fence about it internally.
> For monitoring, we have relied on notebooks this far.
This is pretty interesting approach. Notebooks are a natural environment of Data Scientist and for monitoring they'd provide really good discoverability, flexibility, and interactivity.
Do you use more traditional monitoring that will alert you somehow if a workflow fails, or a workflow hasn't run at all for X hrs/days?
This isn't a scheduler that will trigger 'flows' though is it? If I search "Scheduler" in your docs the top result is a roadmap item, and searching "Cron" turns up nothing.
Is it true that right now that to run a DAG "every day at 3am UTC" requires an external service?
We could provide similar support e.g. for Airflow, if there's interest.
For monitoring, we have relied on notebooks this far. Their flexibility and customizability is unbeatable. We might build some lightweight discovery / debugging UI later, if there's demand. We are a bit on the fence about it internally.