I've been working on a server for Ragnarok Online in Erlang if anyone's interested: http://github.com/aliter/aliter . It's structured similarly I suppose, except it has no monitoring server. Players are represented as gen_fsms, and the servers (login, char, and all the zone servers) can be distributed across various nodes, etc. It uses Mnesia for its database, and the scripting language will likely be lfe or Reia. (It's lfe at the moment, but if Reia evolves enough we may jump over to that.)
Unfortunately as I'm reimplementing a server for a client that already exists, I had to offload some of the time-sensitive stuff onto C as it's simply too slow in Erlang (in this case, walking + pathfinding).
Edit: Also, hot code loading is f'ing incredible for stuff like this. Get on, walk a bit, notice a bug, update it in the code, make:all([netload]) in the shell, bam, fixed. (Though I had to patch stdlib to fix a bug in c:nc that makes make:all([netload]) useless. I've submitted a patch and it's in the pu branch.)
I recently got introduced to Erlang and have a question. How do you unit test your code? I recall reading that the CouchDB author was obliged to use JavaScript for automating his unit tests. What was your experience?
eunit is good for simple functional tests (etap is another option here) but it is not really sufficient for testing things like gen_servers and more complex behavior. OTP includes a testing framework (common test) that does these sorts of tests and is in fact used to test the rest of OTP, but it is a PITA to configure and documentation is rather thin so most people opt for low-level eunit tests and hope for the best when it comes to the more complicated bahaviors of the system...
I'm probably going to use Erlang for the server side component of my next MMO (I created Forumwarz, a web based MMO in Ruby on Rails.) So this was a very interesting find.
It's curious that the author uses TCP. Almost every high performance game uses UDP, as you don't care about lost packets or the order 99% of the time. You just want the latest information.
One reason might be to use Flash for the front end. It does not allow UDP sockets at this time.
EvilTrout - Just to verify your previous statements - I remember when I took networks in the 90s my professor told us UDP was the way to go if reliability wasn't a requirement and performance was important. (less overhead, and of course better latency, etc.) That was the how Everquest successfully implemented it. Many of the EQ players were using 9600 baud dial-up and were very happy with network performance when it first came out. Not to say they didn't have problems at the beginning. But they have long had an awesome hybrid (UDP/TCP) solution that's been tested for a decade.
It's interesting that we've hit a turning point with WoW. I can only imagine how much more pleasant it must be to code using TCP instead of a custom UDP protocol. It also may explain some of the odd network behavior that was "solved" circa 2002 in Everquest but occasionally happens in WoW today.
I think that once you get your abstractions straight, it's not that much more uncomfortable writing for UDP or TCP. I could be wrong, in which case I'd love to hear why.
It depends what kind of game you're creating. For the kind of game we built it was a huge win. Some of the features were deployed in days.
Ruby might be slow, but in my experience as long as you give it a lot of RAM to play with it can give you great performance (we have over 200k registered players). Of course, we also cache data heavily using memcached.
As long as I have you on the line, do you have any nuggets of wisdom for front end development? I was very impressed that the core gameplay appeared to be just Javascript, AJAX, and lots and lots of bubblegum.
I feel like I'm spamming this point sometimes, but I don't think it can be said enough: If you are a language designer and you want to displace Erlang, you shouldn't be targeting "shared-nothing concurrency" or "message passing"; those are merely the door fee. OTP is what you need to replicate/replace/supplant. All of it, including the cross-machine stuff. Believe me, there's room for improvement.
Documentation for starters. There are a ton of hidden gems down there in the weeds, but until you really grok the whole system they are quite hidden. I needed to have a worker query its supervisor for the start args of all of its running children once. The info existed and eventually I found it (sys:get_status called on the supervisor and then pulling data out of some heavily nested tuples), but it was not a part of the standard supervisor calls and eventaully someone pointed me to the sys module; digging in here taught me a lot about how things work, but it is not easy to find if you don't know where to look. OTP is a lot like Rails, there is a steep learning curve but once you really understand how it works you can do a lot of very cool things with the available tools.
You end up with a lot of unfactorable redundancy in the things that implement the behaviors. You end up having to do a lot with records, and the syntax for dealing with them is very verbose. The distinction between handle_info and handle_cast is a bit odd. It's a lot of little things, many of which are actually Erlang-the-language/syntax, that adds up to an experience that is better than anything I know when you're hitting the strengths of the system, but still could use some improvement.
I'm on a laptop with a wide screen and it took me a moment to realize that the navigation links worked, I just had to scroll down to see anything different.
Does anyone know what doing physics in Erlang is like on the zone or area servers? Or do most people just not do physics?
I would think that the physics is what makes things difficult. Of course, if you can think of a game where physics simulations are not as important, an Erlang engine really starts to make sense. At least that is my impression. I'd really be interested in hearing from anyone who has had experience with physics in Erlang.
physics rarely matters for MMOs. The biggest MMOs like wow have almost no physics at all - just gravity and occasionally a single force vector being applied to a perfectly rigid, non-rotating object in a frictionless environment.
It seems that most MMOs, at least of the fantasy variety, tend to ignore physics. Mobiles have maybe three speeds, stopped, walking, and running and transition instantly between them. Jumping is just a pre-programmed arc that may have checks for colliding with other objects. Fall speed tends to be constant and non-accelerating.
However, space-based games would likely require a little more physics to be interesting.
Although I haven't gotten to that part of my game server, I've tried to design it to avoid any sort of physics, because of the potential latency between a client and the server. (I'm targeting mobile users.)
The only 'physics' that need to be calculated is when a player changes speed/direction, and you must calculate the path and detect collisions.
However, to minimize negative effects on gameplay, both the client and the server 'dead reckon' the position of players in their immediate view bubble.
In the client, this is done on a separate thread, on the server, a separate set of processes. Again, I don't expect my algorithms to be cpu heavy, but if they were I'd probably implement them in C and communicate with the erlang processes via message passing.
However, this may not be feasible if the latency is too much, in that case, C/C++ is probably more apt. However, you'd have to implement your own scaling, so I'd rather modify my game design than write the functionality that erlang already provides.
Cool, I've been waiting for something like this to sprout out. I've been working on something similar, haven't yet looked at the code, but I wonder how well it scales as far as number of users, since Mnesia supposedly doesn't support very large tables.
For this reason I chose to use PostgreSQL for things like areas, characters, items, etc. Because it's easier to shard and has a stable track record. However, for soft-realtime data (like area co-ordinates, velocity, etc.) in-memory Mnesia/ETS is the way to go.
disc-only copies are actually limited to 2GB, so available RAM will place a limit on DB size.
From Programming Erlang,
For massive numbers of entries that you want to be able to
access readily, you might be better off using CouchDB, MySQL, PostgreSQL, or Berkeley
DB, all of which have open source Erlang drivers and APIs available. The upper
limit of a Dets table is 2 GB. This means the upper limit of a Mnesia table is 2 GB if the
storage type is disc-only copies. For other storage types the upper limit depends on the
system architecture. In 32-bit systems the upper limit is 4 GB (4 * 109 bytes), and in
64-bit systems it is 16 exabytes (16 * 1018 bytes).
I use Mnesia mostly to store temporary distributed data. If you want to store persistent data with another database you can just replace the implementation module for libsave with your own.
libsave is my library for storing objects to disk.
So far I don't know how many users my servers scales to, with some rough test I can have about 2000 players in the same spot, I recently added a quad tree and with that in place I hope that will increase a lot if players are spread out in the area.
Unfortunately as I'm reimplementing a server for a client that already exists, I had to offload some of the time-sensitive stuff onto C as it's simply too slow in Erlang (in this case, walking + pathfinding).
Edit: Also, hot code loading is f'ing incredible for stuff like this. Get on, walk a bit, notice a bug, update it in the code, make:all([netload]) in the shell, bam, fixed. (Though I had to patch stdlib to fix a bug in c:nc that makes make:all([netload]) useless. I've submitted a patch and it's in the pu branch.)