This is pretty much what Inngest is (https://www.inngest.com/). Runs on Deno as well.

Ingest is definitely nice. Its design is an orchestrator/scheduler where you offload your workers as serverless functions.

The only issue is for background jobs you need to design it in such way to not run in to timeouts.

Which makes it slightly more complex then just having a single executable running periodically.

There is also https://www.defer.run/ which run your code on their infra and don't have timeouts. But they only support node/bun at the moment.

We'll also be supporting long-running services as well in the near future which subscribe to updates from Inngest, rather than get invoked via HTTP.

Currently, to get beyond the ~5 min limit per step, you'd need to deploy to something non-serverless like Fly.io, Render, or your own instance running Express.js or similar.

This is really interesting - we’ve tried really hard to solve some of these with Bacalhau[1] - a much simpler distributed compute platform. Would love your feedback!

[1] https://github.com/bacalhau-project/bacalhau

Disclosure: I confounded Bacalhau

Huh, this is pretty cool.

I took a quick look through the docs and from what I understand you can just submit docker containers to be run on a cluster. The default one is a public cluster. In some way it seems similar to Google Borg.

What I don't understand though is do you have to pay? Who provides the compute?

Maybe possible to build a vercel type service on top of the private cluster where people can just submit their code?

Thanks! The idea is that it’s up to you - we offer a demo network, and there are ISPs who are interested in doing this privately, but the default is to allow you to trivially setup your own cluster (much easier and more maintainable than many cluster providers today - no hate! Just different)

That's exactly what windmill.dev is (https://windmill.dev)

EventBridge + Lambda or ECS gets you pretty far for the last two imo

But then you need to set it up. On small fast moving teams having something that just works from the code is a massive win and I'd rather not cross the infra/code barrier.

You should check Windmill.dev.

I will check out windmill. Though it does look somewhat complex.

One tangentially related is I'm looking for somewhere to just run a python script once a day.

I'll see if I can do that with windmill.

We have been using a self-hosted Windmill instance for running batch processes in Production for a while now. It has been a very nice upgrade from cron jobs for us (adding captured job outputs, status info, ability to re-run jobs, etc).

(disclaimer: Dev Advocate for Airplane)

Seems like that would be a good fit for Airplane's Schedules: https://www.airplane.dev/schedules.

It's pretty much as straightforward as deploying your Python script (and any dependencies with requirements.txt) and setting up your schedule for when you want it to run!

I will try this as well thank you!

Yes, you can do that quite trivially on WM.

I'd add a 6th one:

- workflow system