Interesting. I built something somewhat similar about 2 years ago (not big on blogging so it’s not published - probably end up hurting my career :( )
The gist is using pubsub to send invalidation messages consisting of item ID and time stamp, but also store the latest invalidation message into a set that is polled on every longer time period (10 seconds in my case). Listening to pubsub and polling are done per server that hosts the in memory cache for all items in the cache.
One interesting complication was how to make sure I don’t end up with the wrong (stale) cached item due to race condition between the read and the message passing. Imagine 2 thread, thread A reading and populating the cache, thread B listening to invalidation events. If thread A get held up (context switching, slow network, etc) it can actually populate into the cache after thread B has received and processed the invalidation message. To avoid that I ended up holding the invalidation messages and their receive time in the receiving server for a short duration (20 seconds - twice poll time). I required and in memory cache population to tell me the time from before the read command was sent to the cache. At the time of cache population, if I saw in had any invalidation message marked after the time the read command started I discarded that item instead of caching it. In highly concurrent, highly distributed application this turned to be a must-have.
Also, mainly to placate my fears, I had “panic mode”. If I didn’t see any message (I sent periodic maintenance messages) or if I couldn’t execute multiple polling attempts I dumped the whole cache and wouldn’t let any new item be ca he’d until communication was restored.
Forgot to mention - writes also invalidated the cache locally so maintain “read your own writes” for multi step algorithms. That was crucial for us and I suspect for a lot of other use cases.
The gist is using pubsub to send invalidation messages consisting of item ID and time stamp, but also store the latest invalidation message into a set that is polled on every longer time period (10 seconds in my case). Listening to pubsub and polling are done per server that hosts the in memory cache for all items in the cache.
One interesting complication was how to make sure I don’t end up with the wrong (stale) cached item due to race condition between the read and the message passing. Imagine 2 thread, thread A reading and populating the cache, thread B listening to invalidation events. If thread A get held up (context switching, slow network, etc) it can actually populate into the cache after thread B has received and processed the invalidation message. To avoid that I ended up holding the invalidation messages and their receive time in the receiving server for a short duration (20 seconds - twice poll time). I required and in memory cache population to tell me the time from before the read command was sent to the cache. At the time of cache population, if I saw in had any invalidation message marked after the time the read command started I discarded that item instead of caching it. In highly concurrent, highly distributed application this turned to be a must-have.
Also, mainly to placate my fears, I had “panic mode”. If I didn’t see any message (I sent periodic maintenance messages) or if I couldn’t execute multiple polling attempts I dumped the whole cache and wouldn’t let any new item be ca he’d until communication was restored.