Thanks! SuperChief does one at a time streaming. The library for leader election is just a wrapper around an apache curator leader latch with some instrumentation and logging so we can reuse it in other services.
There's no batching, we have a 1 to 1 mapping of kafka messages to measurements we receive from our api, that could change though over time. Superchief just reads the messages and each message is passed off to another thread for processing.
Hi, I'm the author of the blog post and wrote SuperChief. We chose Java 8 because it's the language we're most comfortable with on the data team and find it easiest to reason about. We also reused some of the existing bolt code we wrote for Storm to run the time series aggregations. It was already written in Java so we just had to make it thread safe. I suspect you could get similar performance with other jvm based languages.
I was interested in looking at Spark for this for a bit but coming from Storm we decided we wanted something stripped down that was more purpose built.
