> Is there a good reason why everyone seems to be using rabbitmq?

it simply does work, has extensive language support (client packages), plenty of good documentation

having worked on & maintained & architected 5 different social backends and failed at 4 of them, here are my key take aways;

data storage: * keep data structure as simple as possible, * do not go with the hypes, stick to old?, proven technologies * have your db constraints as strict as possible in early stages, later you can remove them as for performance improvements * test the key features of your database choices(does transparent sharding really works?), you will see them failing... * you dont need a graph db, you need a graph-like access layer to your data * your >1 month old data wont be accessed/modified at all (mostly) chose your shard key accordingly * keep duplication of your data as small as possible in early stages

indexing: * >1 month old thing might not apply here * ACL can be problematic/hard to manage, try to keep it simple.

queuing: * to me this is one of the most important elements, if you want a simple way to keep every component in sync with others, use event publishing, really. * have your re-try mechanism, dead letter queue, delayed processing in place.

> We can use ActiveMQ, which is the most reliable queuing software. a very bold statement :)

caching: * think your cache invalidation strategy from the beginning * keep your immutable and dynamic data cache separately in your code (at least visually) * try not to mix your business logic and cache code

> We should be ready for anything in the order of billions of queries per seconds. if only you are next/already facebook...

Thank you for reading this and sharing your thoughts.

it is like NATO :)

it is like TL:DR for "Systems Performance" book :)

i had the same history with neo4j

are you using it for production purposes or for your small projects?

if it is for production, how is your read/write performance?

Neo4j wants everything in memory, so the bottlenecks would come after your data-set size outstrips the memory available.

Fast disks are also important for write performance since Neo4j syncs every change - big RAID arrays of SSDs help.

This is one place where Log-Structured-Merge systems can really prove themselves. There is rarely a need to sync on every change as most of them are independent. You can usually gain lots of write-throughput by syncing at the speed the hardware is optimized for and squeezing as many append-only changes in to those logs. At Orly we've spent quite some time looking at ways to deal with these bottle-necks.