Great idea, Erlang is very good for such tasks. Native communication between nodes, very high availability and fault tolerance (native as well) using "actor model"[1] and good performance. As an example, in one of my previous companies we built a message broker, which was able to keep 35K keep-alive connections on Amazon Medium EC2 instance and there still were reserves in CPU and Memory.
Plus it's easier to learn it than it seems first time when you see it. Most of our Erlang-developers came from other languages, but they learned basics very quick: 1-2 days until the first commit on production.
The only one thing, it's not so easy to debug it in the beginning. But it's a question of practice, I think.
Question: on a scale of Go to Haskell, how long on average does it take for devs only familiar with a couple of imperative languages (e.g. two of Python, C, Java or Ruby) to learn Elixir to the point that they can be productive with it?
We haven't measured the actual time. I would say the time learning Elixir has not been a hurdle or anyway limiting factor to the projects we are working on.
We basically switched as a company from writing servers in C#/nodejs/python to just Elixir. Did this gradually during end of last year and this spring.
Though I must mention that all of our developers were already familiar with multiple languages. And since we have multiple projects going on we can transfer knowledge easily. So YMMV.
Sorry, I'm not a person to whom this question was addressed. But in our company we had 3 developers coming from PHP and Python world. They learned basics in 1-2 days and started to commit in production repository. Then in 1-3 weeks they learned it much deeper. But in our case it was easier because we already had high traffic and there were a lot of production examples to learn.
Do they ever have to use an Erlang lib and find the documentation harder because they're Elixir programmers?
I love the Elixir syntax but worry it'd be like Scala: a better way for people who already know Java, rather than a way to use BEAM / OTP etc for people used to modern languages.
Most of my time I write Python, but I'm trying to spend more time with Elixir, and I haven't found using Erlang libs that hard.
There are a handful of rules to remember, if that, and it's reasonably straight-forward. module:func becomes :module.func, the erlang func probably wants char lists rather than strings, atoms are lower-case and should be changed to :atom, vars are upper case and should probably be lowered.
Learning the Elixir language is easy, thinking functionally less so (for me at least).
I don't remember that it was harder actually. We were using Mochiweb (was replaced by Cowboy at some point in time), Chicago Boss and some other additional Erlang libs and people switched from it to Elixir and other way around.
Agree with rossj, there're some rules to remember and then it's not that hard.
I would have to disagree with the other responses to this question. However, I cannot disagree completely. Learning Erlang/Elixir is a subject full of nuance.
Learning the procedural aspects of Erlang or Elixir is very easy. In that sense I agree with the previous responses. Erlang gets a lot of bad rap for the syntax. But, it is really no more difficult that any other language to learn. The syntax can easily be learned in a day. Elixir's syntax can be learned even more quickly because it looks a lot like Ruby or Python. Thus, you can certainly start write scripts in Erlang or Elixir relatively quickly. My guess is that the average, experienced developer could start writing basic scripts in an hour or two. However, those scripts would be entirely procedural in nature, unless they are already very familiar with functional programming and pattern matching.
The first hurdle in Erlang/Elixir is the functional aspects of the language and understanding the power of pattern matching. Both of these require rethinking how you build a program. But, both of these are also incredibly, incredibly worthwhile and powerful. Elixir in particular is a great medium for better understanding these concepts because it is a very approachable language. Getting over this hurdle did not happen over night for me. I rarely used recursion in Ruby or Javascript (there is no tail call optimization). On the other hand, in functional languages recursive functions are the norm. I learned to really love them with Erlang/Elixir and now it frustrates me when I can't use them elsewhere because it has become much easier for me to reason about recursive functions than loops. Overall, functional programming and pattern matching are amazing things to learn but really grokking them won't happen over night.
The next hurdle with Erlang/Elixir is understanding OTP. This is THE incredibly powerful aspect of the Erlang VM. However, it is also not an easy subject matter to approach unless you already have a strong background in distributed systems. I did not. Hence, it took me a while to compose programs that actually used the Erlang VM for what it was built for, OTP. Thus, this portion of the Erlang/Elixir learning curve felt long and slow. I was not used to spawning new processes on a whim and passing messages back and forth. Initially I would send messages only one way without realizing that the appropriate pattern is to view it in a standard server-client relationship where there should always be a request and a response. Eventually I started viewing all the little applications running inside my larger program as self-contained capsules/servers that communicated among each other. However, it took me weeks of really intense study to get comfortable with. But, once you get comfortable with OTP you have a lot of built-in power (people rarely mention Mnesia, DTS, ETS gen_server but they are awesome).
Overall, my experience with Erlang/Elixir has shown me one big thing: pick the best tool for the job and there are a lot of jobs that you don't need OTP for. When you need to build Whatsapp, you should pick Erlang/Elixir. When you want to build a simple web app that only has a couple of end points and much of the dynamism is handled on the front-end, use Go, Node, or my personal favorite Clojure. Erlang/Elixir shine in the context for which they were built. But, they are not built for everything. And, if you only want the procedural aspects of the languages, there is no reason to choose them over Go, Node, Clojure, or Scala...all of which have solid approaches to concurrency.
That was my 1st reaction too ... but hold on. If you design the 2 components to be non-chatty and have robust error handling, then it can be safe to make it location transparent.
In other words, the problem in the past was, we designed it as if it would be used locally, then we naively made it remote. But now, we're talking about designing it for remote use, but then using it locally.
It still has all the other disadvantages of remote services, like consistency problems (everybody always assumes that RPC calls are as reliable as 2 phase commit, and they're not), more and more complex code that's harder to change, less flexibility on all fronts except for the one, ...
Things like distribution are a tradeoff, an optimization : you should never split a monolithic application into 2 communicating parts until you have good reason to do so.
A monolithic system executing a given function is pretty much guaranteed to be simpler than a distributed system doing the same function.
A small amount of microservices is easier on the guy doing ops work for the system, because they can be individually replaced, upgraded or changed (IF the developers put in the constant amounts of efforts needed to make that possible. Every release should be able to deal with past versions of everything it talks to). These guys are also usually the ones doing load balancing and the like, so bonus points for being able to spread them over the network.
Of course, numbers of microservices always grow, and grow, and grow some more. Soon it's not easier anymore on the ops guy, and becomes a constant drag on progress for everyone.
> A monolithic system executing a given function is pretty much guaranteed to be simpler than a distributed system doing the same function.
And what about a monolithic system executing thousands of different functions?
If your app is a simple todo list, sure, go monolithic (but you should still delegate storage to some other tool). If it is much more complex than a todo list, then it is very likely that you have many packages containing many modules developped and maintained by many programmers. And if one glitch on one side of the monolith affects a remote module and functionalities depending on it, or even breaks down the whole monolith, then you should go microservices. Or split your code base. Or do whatever is needed so the thing can still move forwards, if slowly, without breaking apart at each little step.
See , that's the problem with you people advocating micro-services , RIGHT HERE. Your statement is ridiculous and make it hard for people like me to take micro service advocates seriously.
Because we have decades of examples of monolithic apps more complicated than a to do list, but very little time with micro services.
If you honestly believe you can't write robust & scalable monoliths at all, that is an extreme position that is hard to square with many peoples experience.
The problem is that for a monolithic system to be divisible, your team has to have been applying a level of discipline that is not verifiable. It's all on the honor system, and (half jokingly) developers have no honor.
Seriously though, all of the problems breaking the system up will be blamed on people who already left the company, whether they deserve it or not, and the patterns that keep you in a monolith will continue on forever.
The only solution I've seen sort of work for this is to make a single binary, but create modules with extremely clear boundaries. Even so far as compiling them separately if the language is statically typed. But even that can go sideways the moment someone builds caching into the system.
I've come to appreciate why external caches are so popular, but for my money I prefer caching at the HTTP layer. You have to solve all the same problems, but you get one level of cache for free, it's easier for QA to understand, and it puts cache eviction in the hands of mature pieces of code that all implement the same spec (which might account for my previous point).
I agree, the consistency problems should be handled inside each individual services and the layer before all the services should be as simple as normal balancer.
Top-down configuration (as opposed to discovery like UDDI) via something like zookeeper or dns solves this problem. And APIs shouldn't do WSDL either, the code and docs should document the API. It's possible to add so many over-engineered layers of YAGNI flexibility that almost no one can understand what the heck is going on or how to hack on it, e.g., Eclipse plugins.
People could (and did) design robust CORBA and (D)COM objects, but the overhead was too high in some (many?) cases. Overhead in development time, but also run time. If I can pass a const pointer to a bunch of data (if I know the consumer will live in the same memory space), then that's way faster (in development and run speed) than having to marshal that data into a common byte-order and alignment etc., most likely requiring copying, keeping track of who changed what when, ...
Now that I have my old fogey hat on anyway, I might as well state the 'old wine in new bags' feeling I get with articles like this. Encapsulation and separation of concerns is good you say? But inconvenient because there is overhead? So infrastructure to manage dependencies and deployment of sets of loosely coupled components is useful? Whodathoughtthat!
Pass by reference became de rigeur for high performance systems when we agreed that the message passing overhead represented the plurality of computation time.
But pass by reference is shared state, and as we started trying to build systems with threading and for multiple cores the ugly truth about memory coherence on modern processors came out. It was pretty bad. Java and later C had to come up with some pretty odd looking compromises to get safety without throwing away all of the speed advantages.
Yes, you got the idea. We focus on providing developer the "ability to switch or mix between microservice and monolithic styles" so people get flexibility they need for different circumstances (e.g. development vs deployment vs testing).
And then we agreed that location transparency was bad in RMI, and then again in EJB. It's regular as the tides. In a couple years you'll hear everyone talking about how stupid the Cloud is and everyone will be buying their own servers. Then we'll decide that peer-peer is the way and the light.
Didn't Vonnegut have a line about how it affects you to see the same mistakes repeat over and over?
Touché. Every time we hit a new balance in the inequalities we have to re-evaluate all our answers.
I guess what bugs me is not the mind changing (I do that all the time myself). It's that those without a history lesson behave as if this is a bloody revolution, and they have to pick sides. And the people who think switching solutions will magically fix all of our problems.
I don't think the failure of one (or even two) implementations invalidates the concept itself. I see nothing fundamentally wrong with the idea of location transparency (where it matters)
OT but can someone give me an example of some microservices in a normal application? I'm really stuck in the mindset of doing everything as a monolith and have a hard time imagining how I could split things up in microservices.
Really depends on your domain, but reading up on "bounded contexts" and "domain-driven design" will help you in determining which microservices might be named and designed in your area. For example in online retail, seperate microservices for Customers, Products (catalog), Product reviews and Orders might be a feasible distribution of responsibility, functionality and data.
Note that there's always a catch in going from a monolithic application to one with a more distributed nature. Distribution of data requires data synchronization and results in integration challenges, because of the unreliability and higher latency of networks.
Consider a vanilla CMS: Public site where random visitors can view, with an admin part where groups of privileged users can log into and create/edit articles. In a microservice world, I would (and have) split this like so:
* Front end: Perhaps a basic Node.js or Rails app. It's UI only; all data handling is done through APIs to the backends.
* Admin: Completely separate app. Same here, UI only.
* Post backend. This is a data store for articles. It has APIs for creating, updating, deleting articles. It also has a permission system that connects users to the groups that they are part of. This could be a separate microservice, but it's natural to keep it close to the data (also for performance reasons, since every post lookup needs to check if the calling user is allowed to see it).
* Login backend. This has a user database and can do things like accept logins via other services such as Google and Twitter, via OAuth.
* Miscellaneus support services. For example, say you want to support upload images: Make it a microservice that calls into ImageMagick and stores stuff in S3. Or let's say you want to support importing WordPress exports (WXR archives). Make it a service that parses and uploads asynchronously to the post backend.
I have built this as a monolith, and as a microservice system. The microservice platform is much more flexible and clean.
It also allows us to reuse every part for other apps — and this is in my mind the number one benefit that microservices gives you. We've had several new projects completely unrelated to what we've built before, and we have simply reused the microservices we had already created, only swapping out the new front end.
Making reusable microservices is an interesting challenge, since core concepts (things like users and data objects) need to be generalized, and mechanisms built in so that you can make specific features that don't burden the microservices with too many conflicting concerns. For example, let's say that your CMS's public site wants users to be able to "like" individual articles. Instead of building this into the article store, create a microservice which abstracts the concept of "liking". Or let's say you want users to sign up for a newsletter version of the site that sends them a digest every week; again, don't build it in, but make a microservice that abstracts this into "subscriptions", with a registry of emails and what users are subscribing to, and so on. To generate well-formatted emails, simply let the front end provide an API for rendering templates, which this "subscription" microservice can invoke to do the actual rendering. Generalizing it this way makes it easy to then create new types of subscriptions, such as SMS messaging.
Another important point is that every microservice has to be multitenant; a single microservice can host multiple unrelated datasets, and they need to be partitioned in a way that allows the APIs to not comingle the data by accident. It's not rocket science, but it takes some planning and practice to make the APIs feel natural.
Great work - looks really cool, esp for ppl using the Node stack. Really excited by the progress in the space, with products like this, JAWS (https://github.com/jaws-stack/JAWS), hook.io and more.
At StackHut (www.stackhut.com) we're working on something similar, a micro-services based system that's provides a scalable, typed, JSON interface into JS/Python classes wrapped up in containers.
Can't wait to see where this microservices-based path ends up...
Communication between nodes is transparent on the language level, its all message passing between processes.
There is a approachable language with great tooling on top of the Erlang VM [1] and a great web framework [2].
[1] http://elixir-lang.org [2] http://www.phoenixframework.org