We’re experiencing this at my place of work. Our backend stack is Python, and coming from an Elixir/Phoenix background, adding a durable queue to our infrastructure should be trivial but because of the nature of Python, it is all but impossible to run the queue in the same codebase and application server.
Elixir and the BEAM make it so easy and pleasant to run extremely complicated infrastructure in the same codebase and in the same application context. Hard problems made easy.
Interesting, most of my work has been 'across the stack' but for desktop applications. This looks like a better match for me than most other approaches to web applications.
It is a very good point: I did a lot of desktop apps at one point (the largest being a mixture of C# and legacy C++ wrappers), and the first time I tried LiveView it somehow reminded me of WinForms!
Every time I see microservices, or Kafka, or Redis, ... I can't help being disappointed because we seem to be stuck reinventing BEAM/Erlang (and Elixir) 30 years after the fact, except with technologies that are slower, more fragile, more complicated to use and harder to deploy.
Someone in the Erlang community named our processes “nanoservices” as a joke. I thought that was pretty apt though, a gen_server is pretty close. And you can have 3M of those on one node!
> it is all but impossible to run the queue in the same codebase and application server.
I am surprised, both in terms of the difficulty you faced and that you would want to; I have beef with the fact that using Celery with Django runs "worker" process on the application server by default.
It sounds mad but because of the process isolation within the language you can trust running services within your application and message passing can be local or to a different machine in the BEAM cluster. It opens up a whole new way to think about a runtime once everything is an actor model process with a mailbox you can send messages to.
It's much better than having to worry about extra infrastructure, just deploy your app and because it's code you control you can customise what the queue is going in really bespoke and application specific ways.
But why? I have one project that is a single Python codebase that embeds the whole http server, task scheduler, a durable queue implementation, and job processing implementation using those queues all in a single process.
I'm sure you're right that the nature of Elixir makes wrangling all the different services easier but is there something special it does to make embedding easier as well?
Are there any examples online of doing these things with simple (or just not as complex) elixir code compared to using dedicated libraries in other languages
FYI "all but impossible" usually means the opposite of what you think it means - "all but" meaning "everything except". Seen this usage a lot and it bothers me, because both usages are common and theyre the exact opposites.
If I didn't have context clues, it'd be difficult to know which one you mean.
Agreed, to me the meaning unpacks like: "it meets or exceeds every category of difficulty that is less severe than Impossible."
Compare to a disaster described as "the destruction is all-but-total".
That phrase does not mean "you could describe the destruction by any label except total." For example, whatever just happened it definitely not "tiny" or "moderate".
“All but” = “almost” (which presumably historically was “all most”).
Like “all but dead”. Everything except dead, as in as close to death as possible without actually being dead (and certainly not the opposite of dead, as in perfect health).
I'm not sure how it makes sense to claim that an action is "everything except impossible." This is more in the classic sense of "the battle was all but lost" for a battle was practically but not actually lost. You could implement a queue in a Python server, but it would be technically awful to do.
In this context I meant “all but” == “everything except” i.e it is almost impossible. Are you saying it means the opposite of how I used it? I’m not certain what you mean.
Elixir and the BEAM make it so easy and pleasant to run extremely complicated infrastructure in the same codebase and in the same application context. Hard problems made easy.
I miss it.