This is great. I've come across a few articles lately about caches and their effects on performance. It's really interesting stuff. If this interested you, then you might find this article[1] by Dan Luu on improving performance by misaligning data interesting as well.

[1]: http://danluu.com/3c-conflict/

The code for the benchmark: https://github.com/mikeb01/qconsf2014/blob/master/src/main/j...

Can anyone explain why the relative performance of different # threads spreads out again for working sets > 8 MB?

At that point, the performance of his benchmark depends of DRAM access. In modern system, DRAM is very parallel -- a single access is glacially slow but you can have a lot of accesses in flight. It would appear that a single thread of his benchmark cannot provide enough access parallelism to saturate his DRAM interfaces.

When the threads are hitting memory anyway, the cache is less hot, so the contention on the cache is actually reduced.

More threads can probably queue up memory requests a bit faster than one thread which is doing some processing between requests.

I can't explain the precise convergence at the 8MiB mark, though.