HHVM and Dart seem to be the two new fast performers in town showing impressive performance in some tests. JS has been falling off the charts compared to some of the first rounds, but still a good option performance-wise. C# keeps sucking badly. I miss Nimrod/Jester, I always wanted to see it in the top 10.
These benchmark tests for C# are run against MySQL or PostgreSQL on Linux. In the Fortune 500 setup you're probably connecting to SQL Server or Oracle in the back end for which Microsoft and DB vendors have optimized OLE-DB drivers.
That, and JSON serialization on .Net using default MS serializer is super slow. Everyone uses JSON.NET or another faster serializer in the real world.
We have SQL Server tests but they were not included in this round. They were last in Round 7 and we'll include them again in Round 9. Here is SQL Server in Round 7:
Also, we have a JSON.NET test implementation thanks to community contribution. It's test #138 and named "aspnet-jsonnet" as seen on the following chart of C# tests:
Just time. We were already two weeks late on Round 8 due to a host of other issues. We'd like to do one round per month if we can get our routine ironed out.
We'll make it a priority to get them run in Round 9.
Please don't be annoyed by my following comment as I appreciate your effort and like the benchmarks very much (as a hacker - as project leader I'd prefer the "enterprise frameworks" to perform much better :-) ): You should really avoid publishing incomplete benchmarks. They don't do justice to both, the left out and the included.
> You should really avoid publishing incomplete benchmarks.
If we had followed that advice, there never would have been a round 1. I was so uncomfortable with the idea that we'd be publishing surely-flawed benchmarks of frameworks we didn't really understand that I requested to be taken off the project (prior to round 1). It was only after seeing the post-round-1 discussions and the flood of pull requests that I realized I was wrong.
These benchmarks are always going to have flaws. I think it is better for us to regularly publish our best attempt than to try for perfection.
How about those results under the plaintext benchmark run on windows with no db access to mysql or whatever? Still very slow. The low level libraries for http/disk/etc for C# on windows or linux are simply not set up for performance in general and that's what this benchmark is reflecting. You're dismissing these results a bit too quickly and defensively, I think.
Those numbers might not match up to other frameworks, but they are by no means slow. ~29k requests per second (standard ASP.net) equals 2,505,427,200 requests per day.
That's far more capacity than anyone needs and if your site does ever reach the point where 2.5 billion people visit it per day then you can just put another box up and double your capacity to 5 billion requests per day.
This doesn't equal out to 2.5 billion people a day. One requests does not equal a unique visitor. For most cases that doesn't even equal one page load. I don't know where the 29k/s number came from but I guess its a peak load from one of these bench marks. It isn't realistic to expect a server to be consistently pegged at 100% 24 hrs a day. The real number is going to be a tiny fraction of that
Sure, as long as your website is just a tiny piece of plain text, and all your requests are distributed perfectly evenly over the course of the day, and every person only requests a single file and then leaves. But since none of that is even remotely close to realistic, your numbers are not either. Yes, 29k req/s is quite slow for a "serve a tiny static file" benchmark. No, you can not handle a billion visitors a day on one server with such low performance.
Probably because you threw Fortune 500 sized budgets and hardware at them. This is only really relevant to people who need to maximize limited hardware.
Because performance can largely be displaced to outside the app code itself. Things like cacheing, load balancers and large amounts of front-ends make the optimization differences of these platforms largely irrelevant for large projects, and especially large companies that don't mind throwing more money to increase the amount of front-end servers.
For smaller projects, or for companies / people with tight budgets, these performance tests matter more, though the biggest wins still lie in cacheing and load balancing, not in platform efficiency. This can depend on the nature of the application though. Some have tons of cacheable content, some have tons of dynamic content.
>saving tens of millions of dollars is largely irrelevant
Why do people seriously say nonsense like this? Why do you think facebook spends so much money on hiphop/HHVM? Because yes, performance certainly does matter. Slow languages and slow frameworks cost tons of money.
Well, it's not irrelevant, but it's common practice. Of course bad practice. Seen it myself. Facebook and others might be able to choose the better approach but a lot of badly managed companies / projects choose the last resort solution to through out millions of dollars for hardware as they're incapable of fixing the problems properly. Those projects are politically screwed, so they often sell their solution to buy expensive hardware as success. Sounds ill, I know ...
The evolution of a project as I've seen it at large companies:
1. Build slow app in high level language, build out your infrastructure with cacheing and load balancing, CDNs, etc.
2. See what is slow / can be optimized in the current language.
3. Re-implement critical pieces in a lower level, more efficient language.
Facebook is building HHVM to eliminate step 3. You can have a bunch of people continuously re-implementing critical pieces, or you can have a smaller bunch of people make the higher level language more efficient once and have all your projects, current and future, benefit. And you also save money from the inefficient language.
I'd like to see the numbers on how much, say, using Ruby with all the optimizations - load balancers, cacheing, etc. will cost you over using Java - also with optimizations. I'm not convinced by a random person's statement that it is that huge of an impact, especially considering factors like finding programmers for optimized languages, productivity differences in programming in the different languages, etc.
Yeah, finding java programmers is super hard. Not like there's literally hundreds of times more of them than there are ruby programmers or anything. Facebooks is building HHVM because it is too late to do it right in the first place. If they started with reasonably performant language they'd have saved millions on servers, plus millions on developers writing HHVM. You are posting in a discussion about the very numbers you want. Go look at them.
That is a generally meaningless statement. You may have a hundred front-end and application servers to service a hundred users. Your users may be accustomed to tolerate very slow service times (e.g. most corporate systems inject several-hundred millisecond delays for the most trivial of operation). Etc.
I've built plenty of .NET-based services, and generally it was very powerful hardware serving a relatively small user base, and where expectations were much less demanding. And that's perfectly fine if the other benefits of the system (tooling, integration, etc) works for the implementation.
For someone building a startup on a shoe-string budget, though, it has to be foreboding seeing such poor metrics when that directly translates into considerable additional hosting expenses.
Unfortunately, I still have not had enough time to improve Jester (or this benchmark) so its performance is still at the stage that it was on in the previous rounds. Hopefully this will change soon. Of course help is always welcome, so if you want to see Nimrod higher in the results then please help us improve the benchmarks!
something to show how important the VM is. PHP managed to jump near the top simply because Facebook decided to bump money and wrote HHVM. Python has PyPY.
