Can anyone who has done it both ways characterize the tradeoffs vs. using amqp (say, rabbit) as the bus? I've been excited about this feature in redis, and this is a very interesting use case for it, but when I read the part toward the end regarding consumer checkpointing to prevent premature log truncation I was thinking message queuing acknowledgement semantics basically deal with that for you. I'm sure its more costly, but if you need to know whether something's been consumed then you need to know.
The point of this kind of log architecture is that producers never need to know if a consumer has consumed something, because the log is designed to allow as many consumers as you like to reliably work through the log from any given position.
As a producer, you just need to know that your message has definitely been delivered to the log. What consumes it is none of your business.
> The point of this kind of log architecture is that producers never need to know if a consumer has consumed something, because the log is designed to allow as many consumers as you like to reliably work through the log from any given position.
Right I think we're saying the same thing. In the article checkpointing was introduced to enable safely managing the size of the log, i.e. it appears that in this use case _somebody_ needs to know whether the consumers are caught up in order to adhere to at-least-once semantics. It just seemed to me that if you reach the point where you're rolling your own to solve that problem there are other possible solutions.
I haven't read the linked-in piece yet but I intend to. It was also linked in the article. Thanks for linking it here.
