Wouldn’t it be valid to consider Kafka/RedPanda a bottleneck and another point of failure that may delay data getting to a destination?
In some cases, the performance, efficiency, and reliability gains from caching and consolidation make sense.
But, I’ve seen enough poor architectural decisions and lack of architectural oversight result in use of various log streaming, cloud messenging, app monitoring, object DBs, etc., all discounting the request overhead in time and traffic, points of failure, and overall complexity for some false sense of scalability enough to where things that seemed cool ten years ago make me physically sick now.
What are some questions to use to help determine whether Kafka/RedPanda actually make sense to use, without having to first baseline, then implement, then compare request time, reliability, and data freshness to gauge whether it was worth it?
BTW- I think there are valid cases for using it and appreciate all of the work!
If you use transactions, we don't support it yet. We are in active development here.
Notice that folks love speed but the operational simplicity of having one binary/fault domain makes a lot of our enterprise users use the tech.
Last is if you like the product direction which is to my knowledge fundamentally different from the other engines out there. WASM in particular solves around 60% of all streaming work we see in the wild. It is effectively good at one shot transformations (gdpr, simple enriching, simple connection to downstream system like elastic, etc) as well as tiered storage.
Think the idea was to build something as easy as nginx - apt-get install redpanda and et voila
I hope to continue to focus on the developer experience.
In some cases, the performance, efficiency, and reliability gains from caching and consolidation make sense.
But, I’ve seen enough poor architectural decisions and lack of architectural oversight result in use of various log streaming, cloud messenging, app monitoring, object DBs, etc., all discounting the request overhead in time and traffic, points of failure, and overall complexity for some false sense of scalability enough to where things that seemed cool ten years ago make me physically sick now.
What are some questions to use to help determine whether Kafka/RedPanda actually make sense to use, without having to first baseline, then implement, then compare request time, reliability, and data freshness to gauge whether it was worth it?
BTW- I think there are valid cases for using it and appreciate all of the work!