Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

"Also, I've read some comments where people mention "we don't need redis", "we don't need workers" everything is so much easier. That was our thinking at first. But then you realize on deployments you will lose your cache, or your background jobs, etc. So you have to persist them either in mnesia or in the database. At that point you're just reinventing your crappy undocumented and untested version of delayed_job... Most of what you get from elixir in terms of redundancy, high availability, etc you can have that anyway from kubernetes, heroku or any PaaS.... you will need more than 1 server anyway, so..."

I used Erlang for 5-6 years in a production environment in a deployment that wasn't the world's biggest, but did seriously need clustering because we needed more than one machine to handle the load, not just redundancy. This comment leads to my core observation about the entire ecosystem for anyone considering using anything in the Erlang ecosystem.

Erlang started in the late 1990s. Joe Armstrong was brilliant, and I would imagine was also surrounded by some other very smart people whose names I do not know. Despite what I'm about to say, let nothing I say be construed as disrespect for either the language or the people making it.

Let me set some context here. C++ ruled the day, and still looked to inevitably replace all C. Python, Perl, and the entire dynamic scripting language category had just been invented. Java was in the news, but not on your computer yet. Haskell... pre-monadic IO... had just been standardized somewhere in here. Threading was possible but was every bit the nightmare it has been presented as, and you had to be careful spawning more than single-digit threads because of the resource consumption. Open source was just beginning to be A Thing... that is, it had existed in its embryonic form for decades, of course, but it was just beginning to cohere into the Linux world we know now. Machines were still measured in the hundreds of megahertz and the single-digit megabytes of RAM.

Erlang was a far more radical departure from anything else in this era than it is today. There was a lot of academic work on this stuff, and there was a lot of very specialized high-end work on multiprocessing, but it wasn't being done by "mere mortals" and it wasn't very simple. Erlang's designers looked out into the world, and what they saw was a huge jungle, where what maps we had had huge regions that just said "Here There Be Dragons". And they started in, hacking and slashing and slicing their way through, guided by a few intuitions and a whole lot of engineering firepower.

And they largely succeeded in creating a settlement in the wilderness. They grew and managed to pave over the hack & slash paths into roads, built a community, built a pretty incredible VM, built an ecosystem around them. Let this accomplishment not be underestimated; this is a land that had eaten many others in that era.

However, sitting here in 2021, this is no longer a wilderness. Much of the area has been razed, highways driven through it, McDonald's by every exit, and millions of people living here in various bustling metropolises.

And I think what we've found is that where Erlang plopped itself down is OK... but not more than that.

As a consequence of its isolation, Erlang bundles a lot of things into itself that just didn't exist back in the day. It bundles in messaging passing and a network-aware message bus, almost literally decades before anyone else even thought of a "message bus" as a distinct product segment. (i.e., they existed, you could find academic discussion and some early passes at it, but it wasn't really a distinct category yet.) It has a threading environment. It has these "supervision trees" idea which are cool. It has immutability at a time where Haskell was even crazier than it is now. It has a mechanisms for bundling and distributing applications. It has this entire alternate-reality ecosystem in it. But...

... in 2021, none of these are best-of-breed anymore. Many of them are deeply quirky. Immutability is cool, but we only needed to not be able to pass mutable references on the message bus, not be fully immutable within an Erlang process. I believe Elixir fixes this one. A modern message bus going between processes gives you this whether you like it or not because it also can't carry mutable references between OS processes and systems. Having a message bus integrate into the language is a big mistake, because it makes it hard to hook up anything but Erlang nodes to Erlang. (I say "hard" and not "impossible" because I know it can be done, but it's hard.) A modern message bus like Kafka or the dozen other choices doesn't impose an implementation language on you, nor does it impose that implementation language being the only one. Process restarting is a necessity for production-grade systems, but we've settled to a large degree on OS-level handling for restarts, and within a system, there are easier ways to accomplish the goal than bring in the entire Erlang supervision setup. Mnesia, a clustered DB was a neat idea, but in 2021 is so poorly featured and unreliable it doesn't even qualify as one anymore; nobody in their right mind would bring up an Erlang cluster just to back some other language's code to Mnesia as a database. Pattern matching is cool, but not cool enough to justify all the other issues that come with it, and proper use of other language functionality is often good enough anyhow. (I don't tend to miss it; I tend to properly use some form of polymorphism instead. Trivial pattern matching is replaced by this, and to be honest, non-trivial pattern matching is probably a code smell if not an antipattern; if you're reaching three levels down into a data structure, you know too much about that data structure!) Modern Erlang performance is meh; it used to have the clear advantage when dealing with lots of sockets but now there's a lot of things that can comforably exceed it. Its type system is an annoyance even by dynamically-typed standards, if you like static types than just stay away. (They will cite Dialyzer, but it isn't anything remotely resembling a replacement.) And if I reviewed the docs I could find a few more of these, plus I find there's a lot of cases where Erlang has solutions to problems only Erlang has in the first place, e.g., "gen_server" is neat, but the entire gen_server set up, with its initialization phase and its three proscribed ways to call it and the need for an explicit "gen_server" module type are largely the creation of Erlang in the first place... in other languages you generally get the functionality packed up differently but it's all there without having to exactly match what Erlang does.

So, in 2021, the problem is that going into the Erlang ecosystem tends to lock you into a whole package of things that vary from "not best of breed but mostly OK" to "significantly inferior to the modern alternatives". All it really has is very nice integration of its not-best-of-breed stuff, but even that kinda becomes a trap after a while, like when you realize you need a real database after all, then maybe you have to integrate with another message bus so you can integrate with non-Erlang code, and, you know, a couple more cycles around that loop and you'll really regret having chosen Erlang.

One thing that I think it has going for it, and why it's probably still got a cult following today, is that if you're a relatively new developer, or not experienced in network programming, it is a heck of a trip when you get into it, because the tutorial will introduce you to half-a-dozen ideas you've never seen before. Cross-server message buses? Amazing! A network database with an easy API? Amazing! Spawn a million processes on a commodity box? Amazing! I think this explains a lot of the appreciation it gets today. But no longer are any of those things unique. It's just that the normal developer lifecycle will tend to encounter those, one at a time, over the course of years, instead of having them all thrown at you in one amazing, mind-blowing language introduction.

But in 2021, you can do not just "better" than Erlang for all of them, you can do much better. Except that whatever you put together will be something you had to put together, and it won't be quite as nicely integrated. But it will allow for multiple languages, it'll allow you to better integrate with the direction modern ops is going, it'll scale better both internal to your language and in terms of performance, and you'll be better able to hire for it.

The Erlang ecosystem was brilliant and far ahead of its time. I honor it as an incredible pioneer and recommend, even after everything I said, that anybody thinking of writing some incredible new language spend some time in the Erlang ecosystem to see some of what is possible with that sort of integration. But I can't in good conscience recommend it, in either its Erlang flavor or its Elixir flavor (or its Lisp flavor or anything else) to a modern developer. Everything Erlang does is much better done now by other things, minus the language/ecosystem lockin. This is not because Erlang lost, it is because it to a large degree won. It blazed a trail, and we all now agree, it was a trail very worth blazing. But the next waves of settlers & builders ultimately have created something better.



> But in 2021, you can do not just "better" than Erlang for all of them, you can do much better.

Really appreciate the effort that you've put into this post but it's 99% saying you can do better but without examples of stacks that would be better. For those who don't have your knowledge could you provide some examples?


The core insight of Erlang is that having lots of little processes communicating over a message bus is a great way to design code. It also proves by demonstration a statement that many programmers, especially in the past decades, would have found hard to believe, that you can structure code as a whole bunch of relatively small self-contained services. Many programmers, perhaps even today, would not believe how far you can get with such a model, thinking that fairly massive monoliths are still really the only way to scale up.

Rather than specific examples, let me provide terms to google, as these are now such rich spaces that even a list of examples would be impoverish. Google "message bus"; there's half-a-dozen solid production options you can deploy yourself, and all the major clouds have at least one option, often more. Pervasively use a message bus in your architecture and you can't hardly help but end up programming in a very Erlang-esque fashion.

"Database"; no longer is an SQL database over a socket your only option. Some databases even combine a bit of message bus functionality, like Redis, or Postgres.

Restarting and reliability: Process-level restarting of much smaller processes has become popular with things like Kubernetes, Docker, and even just plain ol' Systemd. (This is one place that I will assert that systemd is better than the init.d-based system, which had a solid story on how to start processes but a very ad-hoc one on restarting failed processes.) Internally, consult your favorite language for monitors or restarts within a process; I've got one for Go called suture. Getting an idiomatic restart into a language depends on tons of details and I don't have a good sense of the options across dozens of languages. Note this only really matters if your language and program is handling a lot of things at once in a single process, which is not always the case.

That's most of what would matter, I think. Most of the rest of Erlang like "pattern matching", well, do whatever they do in your language. Which may be pattern matching.


This comment sold Elixir for me more than anything. On a small team, each of those are massive headaches. Using Elixir I can have all of that with a unified vocabulary, documentation, deployment, syntax, repository, etc. Sure, each might be slightly worse that the best thing on offer today, but I don’t have to glue them together.


And you can swap a built-in feature for an external component later, if the need arises.


Sure, you can achieve what Elixir does by combining a bunch of different components replicating its features, all with their different execution schemes, configuration syntax, deployment constraints. That's the whole point of Elixir, it gives you an integrated environment to do these things with consistent vocabulary, documentation and deployment strategies.


But Erlang provides something that Kubernetes/Docker/Systemd isolation does not: fine granularity / per request isolated processes. Is the difference between a code bug killing a single call in a VoIP system vs killing an entire OS process with hundreds of them.


Do you mean actors? I kinda hate how they’re always referred to as processes.

https://github.com/actix/actix https://github.com/akka/akka


The actor terminology isn't used in any official sources since Erlang processes are not quite "actors" in the actor model sense. They were inspired by OS processes.

http://erlang.org/pipermail/erlang-questions/2014-June/07979...


Thanks - that's very helpful.


I'm confused by the extent to which you believe the things that Erlang/OTP and the BEAM do have been entirely replicated by other ecosystems. I think you're quite right that distributed systems engineering and threaded programming as a whole has moved in the direction of Erlang on a lot of fronts, but I don't really see the parallels (no pun intended) at the language level. Most popular languages still depend on cooperatively scheduled threading models (even async/await depends on the dev inserting yield points), while the BEAM's does not.


"I'm confused by the extent to which you believe the things that Erlang and OTP and the BEAM do have been entirely replicated by other ecosystems."

One of the reasons I make a bit of an accusation that the people bedazzled by Erlang/Elixir may be not very experienced is that a common take away from the community I have repeatedly noticed is the belief that if another language doesn't have exactly what Erlang has, warts and all, it doesn't have what Erlang has. Thus, I see in many other languages various attempts to port the exact thing Erlang has out into that other language, when in fact the other language already has solutions to that problem, even if they aren't exactly how Erlang solves it. [1]

While I personally exceedingly strongly agree that the Erlang-like threaded approach is vastly superior to manually explaining the asynchronous-ness of your code to the compiler, it still remains the case that "async/await" largely solves the problem of handling millions of threads at a time in those languages that support it, and even if that particular element of it is inferior to Erlang, it will be compensated for by the wide variety of other things that are superior. Just because it isn't exactly like Erlang doesn't mean it's not decent solution. Moreover, my exceedingly strong agreement is also, like, my opinion, man, and there are plenty of people who disagree with me and consider async/await superior, and they would account this as simply being better than Erlang/Elixir.

In the 1990s and the 200Xs, Erlang had solutions to problems that no other language had solutions for, or where Erlang's solutions where clearly head and shoulders above. In 2020s, Erlang no longer has anything that it uniquely has the solution for, even if the exact Erlang solutions don't exist elsewhere.

[1]: I wrote about this recently at https://news.ycombinator.com/item?id=26833616


>One of the reasons I make a bit of an accusation that the people bedazzled by Erlang/Elixir may be not very experienced is that a common take away from the community I have repeatedly noticed is the belief that if another language doesn't have exactly what Erlang has, warts and all, it doesn't have what Erlang has.

On one level yes, they have constructs that you can use to accomplish some of the same goals as the tools Erlang provides, but some people (me) like the specific programming model that Erlang provides for dealing with these problems. The concurrency characteristics of the systems I hack on every day would be much harder to model without GenServers. It's not particularly surprising that Erlang is not the only way to solve a problem in software engineering, there is no silver bullet.

I've written much more C# than I have Elixir in my time as a developer and the fact remains that I prefer functional programming and I like using OTP. I don't see what the "wide variety of other things that are superior" are in languages that aren't Erlang. The ecosystem is stronger and the tooling is better, but those are not intrinsic to the design of the language. Tooling kind of is, with static languages, I suppose. Lack of static types are the one thing that bother me a lot about Elixir.


"The concurrency characteristics of the systems I hack on every day would be much harder to model without GenServers. It's not particularly surprising that Erlang is not the only way to solve a problem in software engineering, there is no silver bullet."

Erlang was at one point the only way to get this sort of structure, though.

Now it isn't. Multiple languages like Go have lightweight coroutine-y type things; if the isolation isn't perfect, you can fix it with programming discipline. There's also Pony, and Rust which has its own thing which is in its own way far stronger than what Erlang offers. Async/await languages are very amenable to setting up lots of little process-like things, to the point that arguably it's the only sane way to structure your programs in such languages. There is a real change in Erlang being almost the only thing on offer, to merely being one choice of a whole bunch and no longer being impressive enough to justify the lockin. (Not that Erlang uniquely has lock in; all languages have lock simply by virtue of the fact that once code is written in X you can't just change it to Y one day. But it's not worth the lock in anymore.)


>Erlang was at one point the only way to get this sort of structure, though.

It still is though! Isolated processes with asynchronous, dynamic message passing requires really heavy libraries in most other languages. Rust is completely different, I can't fathom why you'd compare them in this way.


Because as I've said, I'm not comparing solutions, I'm comparing problems. Rust solves the isolation problem in a completely different way. Go also solves the isolation problem by enforcing it at a language community and best practice level, rather than rigidly enforcing it in the language. Now, I personally wish it had something more like Erlang or Pony, but, at the same time... it's doing fine without it and completely satisfying my Erlang use cases in real code, better than Erlang did. Not perfectly, but fine. (I rather prefer channels as a default to Erlang's message passing, which in both my opinion and experience has the "unbounded buffer" problem in that messages can build up in a queue forever.)

Message buses solve the message passing requirement. You can get a whole range of solutions from "just sling vaguely JSON stuff around" to "here's a whole strongly typed protocol", with a whole array of options on persistence, clustering, language support, etc., a whole array of options around how synchronous they are and what guarantees they provide, rather than Erlang's one exact solution,

You're kind of demonstrating my exact point here... it isn't that the exact Erlang solution shows up in other languages, the point is that the problem space that Erlang covers is now covered in plenty of other languages, in all sorts of ways, many of them better for some task than Erlang's one point in the space that was selected very, very early in the exploration process. This was not the case 20 years ago. It is now. And I don't particularly think Erlang's coverings of the problem space are the best. People not already deep into Erlang aren't evaluating whether or not some solution is exactly like Erlang, they're evaluating whether the solution is a good solution on its own terms, and once you remove the "must be exactly like Erlang" clause there are a lot of better choices for every element of Erlang, except arguably, the integration of all of them together.


I understand your argument, I guess what I'm trying to say is that I don't think these arguments are unique to Erlang, you could make these arguments about the use of any language to solve almost any problem, outside of kernel programming. I choose tools based on how they fit with how I want to solve problems.


> you can fix it with programming discipline

You can write distributed coroutines in assembly with enough programming discipline. What is interesting in a language is not what it lets you do, it's what it prevents you from doing.


> Immutability is cool, but we only needed to not be able to pass mutable references on the message bus, not be fully immutable within an Erlang process. I believe Elixir fixes this one.

In fact Elixir does nothing for this. It is true that in Erlang, you cannot rebind a variable and must use a different variable name for every assignment. In Elixir you can rebind the same variable, but the data is just as immutable as in Erlang, and the rebind is only visible in the current scope, which trips up new programmers not used to it.


Erlang started in the late 80s and saw production use roughly 95


I think that it would be helpful if you provided examples about those things, good and bad. As it is it's mostly "erlang's" bad - perhaps if you offer the counterparts to what would be a better pick we could have a discussion - and then perhaps see how far the tree of dependencies goes on each side? And also what you would loose by picking up those solutions?

Just taking on some random stuff:

> Trivial pattern matching is replaced by this, and to be honest, non-trivial pattern matching is probably a code smell if not an antipattern; if you're reaching three levels down into a data structure

If the data structure is being accessed in a module to deal with it specifically then that is part of the knowledge of the module. And you can't escape that with or without pattern matching, since you'll have to codify those things in code anyway, be it long chains of ifs, switches, or whatever, you'll always have to write the code with the same level of knowledge about the data structure to express the same conditions?

> Modern Erlang performance is meh;

This I'm not sure. I've seen libs written in elixir beat libs written in C++ and they had to start a VM to be run. Obviously there will be plenty of stuff that a low level language can do more efficiently and faster I don't think anyone disagrees with that - but I think sometimes people sell erlang and elixir's speed short.

> but the entire gen_server set up, with its initialization phase and its three proscribed ways to call it and the need for an explicit "gen_server" module type are largely the creation of Erlang in the first place

I haven't seen any language with the concept of an independent program, that has a synchronous interface (mailbox), but can be modelled entirely asynchronously, and can't block whatever running loop. (and I read your other reply and thread). There's also a reason why some things have a particular way of being set up inside the beam, and that's mostly related to the guarantees it provides and with the fact that processes are transparent across nodes. If you take out the impossible to block scheduling and network transparency you'll be of course able to spin up something similar to a GenServer without the same "ritual" - but it no longer has anything to do with the former perhaps except its API?

> A modern message bus like Kafka or the dozen other choices doesn't impose an implementation language on you, nor does it impose that implementation language being the only one.

Given that you can have any socket, tcp or not, and easily handle it, this doesn't seem to be a fair point. On kafka or any other message bus the implementation language will be that of the sources and sinks communicating with kafka? Or you just write kafka and the other programs wake up as insects in a bed?

> Mnesia, a clustered DB was a neat idea, but in 2021 is so poorly featured and unreliable it doesn't even qualify as one anymore; nobody in their right mind would bring up an Erlang cluster just to back some other language's code to Mnesia as a database

I haven't used mnesia at scale and indeed heard people complaining about it, more than once so there's probably some issues there - having said that, in terms of similar db's (KV based and not SQL,etc) even db's like MongoDB and such need expensive and usually paid managing solutions right? And can still loose data unless you really cover your redundancy?

> But in 2021, you can do not just "better" than Erlang for all of them, you can do much better.

Could you tell us succinctly how? As a, usually, solo developer I would be really interested in knowing, and I'm not asking sarcastically.

I also don't understand how it's difficult to integrate Erlang with other things, it basically has a standard lib very accommodating of sockets and inter process communication through STDIN/OUT, with effective monitoring on top.


"If the data structure is being accessed in a module to deal with it specifically then that is part of the knowledge of the module. And you can't escape that with or without pattern matching, since you'll have to codify those things in code anyway, be it long chains of ifs, switches, or whatever, you'll always have to write the code with the same level of knowledge about the data structure to express the same conditions?"

There are other options. In object oriented design, using the loose definition of "anything with 'methods'" (thus, including Go and Javascript even if they don't have "inheritance"), if you want to know something, you ask directly, using a method for that case. It is the difference between "account.getBill(lastMonth)", where you load the logic into the account object to fetch bills from different times, and pattern matching where you write into the code taking the bill deep assumptions about how the account is structured, such that when refactoring the account you have to go change all its consumers. Note I am only criticizing deep pattern matching here; pattern matching in general I like, even if I can live without, but in Erlang terms, if your patterns look like

    [thing, [_, _, {something, X, [Y, _, Z]}]]
you've gone off the rails.

"I've seen libs written in elixir beat libs written in C++ and they had to start a VM to be run. "

There's almost always some microbenchmark a slow language can beat a fast language in, but that doesn't make it a fast language. You might want to dig into that "elixir lib that beats C++", because either the C++ was really bad or the elixir lib is actually written mostly in C(++).

"I haven't seen any language with the concept of an independent program, that has a synchronous interface (mailbox), but can be modelled entirely asynchronously, and can't block whatever running loop."

Yeah, but when you're talking about a 30-40 year programming language, that's not a necessarily a compliment. Nobody else has seen fit to exactly copy it, because we've found better things to cover the problem space than that.

"Given that you can have any socket, tcp or not, and easily handle it"

There is a lot more to being a message bus than just having a TCP socket.

"I also don't understand how it's difficult to integrate Erlang with other things, it basically has a standard lib very accommodating of sockets and inter process communication through STDIN/OUT, with effective monitoring on top."

That is not "easy to integrate with", that's table stakes. What I'm referring to being hard to integrate with is how you tend to be stuck in the ecosystem. It has monitoring, sure... but in 2021 it's a very quirky monitoring that doesn't integrate with any of the rest of what has developed. You can, in theory, connect to an Erlang cluster without Erlang, but in practice the requirements of being "an Erlang cluster" are so specific, with its own quirky term format that isn't quite JSON or quite anything else, is so difficult as to not be worthwhile. Once you've taken the time to speak to a non-Erlang message bus, use a non-Erlang database, wrap your Erlang code in K8S or some non-Erlang manager, use Erlang to hit HTTP APIs, speak to non-Erlang monitoring systems... why are we using Erlang again?


I still don't understand the point about the deep pattern matching. In OOP you'll have an object that encapsulates all that knowledge, and in functional programming you'll have a module that encapsulates all that knowledge, clients of both the object in OOP and clients of the data structure in FP will need to have the same knowledge to deal with it - once you need to serialise the information. I just don't find the argument compelling, in my reading it would be akin to saying, it's a code smell a class has all this implicit knowledge of the object it's modelling (assuming the deep pattern matching in the module dealing with transforming it).

> There's almost always some microbenchmark a slow language can beat a fast language in, but that doesn't make it a fast language. You might want to dig into that "elixir lib that beats C++", because either the C++ was really bad or the elixir lib is actually written mostly in C(++).

Usually it's microbenchmarking that doesn't flatter elixir/erlang - and stampede problems where you can get away with just brute forcing - eg. parsing directly to output a bunch of files where the output is a file(s) - but there, if you want then to build usable metadata through all of it things change fast - writing a correct program in any of the fast languages that has the same ergos for keeping usable information throughout and is at the same time easily changeable, can interleave information in parallel with pretty amazing monitoring, and doesn't require years of training, is quite a different thing.

I know it doesn't use C++ because I wrote it and I know how much more it does as well, though I'm not here to convince you.

> Yeah, but when you're talking about a 30-40 year programming language, that's not a necessarily a compliment. Nobody else has seen fit to exactly copy it, because we've found better things to cover the problem space than that.

Well, sure, I just think it should not be written as "cover" the problem space, is more "making holes" in the problem space.

> There is a lot more to being a message bus than just having a TCP socket.

Given that your argument was:

> A modern message bus like Kafka or the dozen other choices doesn't impose an implementation language on you, nor does it impose that implementation language being the only one

I'm not sure there's any more esperanto than tcp/sockets, which comparing to any other language are a joy to write, use, monitor, and deal with.

> You can, in theory, connect to an Erlang cluster without Erlang, but in practice the requirements of being "an Erlang cluster" are so specific

This sincerely... How do you connect to other clusters in other languages (if they're even able to set up meshes)? You don't. How do you solve it? With clustering solutions. How is it that a runtime that can be put inside those same clustering solutions, by the same process, but has a zillion more functionality can be worse than those other solutions regarding that?

> Once you've taken the time to speak to a non-Erlang message bus, use a non-Erlang database, wrap your Erlang code in K8S or some non-Erlang manager, use Erlang to hit HTTP APIs, speak to non-Erlang monitoring systems... why are we using Erlang again?

Yeah, because they all speak something else than http/tcp and we know erlang is not really made for handling sockets and can't do http.post().


Can't upvote this enough




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: