Cool, would be nice to see more competition for Conduktor. I have worked with a few “Kafka is a database” people in my career and even if you agree with them, it’s still a terrible one from a tooling perspective. Having to debug Kafka issues on topics with almost any volume of data is slow and frustrating, either with a GUI or command line/REPL tools. Your only hope (if you absolutely have to wrangle Kafka in the core of a project) is heavy testing with the test driver from the start and hoping to avoid getting into too many knots. It’s interesting to note that Conduktor’s more enterprisey features are in this area now.

> Having to debug Kafka issues on topics with almost any volume of data is slow and frustrating, either with a GUI or command line/REPL tools.

Agreed. The scripting feature in Franz is born out of my frustration with some of this stuff and seems like a promising route for quickly visualizing and interacting with what’s happening on a topic (though it’s early days and I’ve got lots of work to do until it becomes truly useful).

i’d love to hear some Kafka as a database success/horror stories. What’s the general consensus?

It works well with master data [1], i.e. semi-static data of limited volume that is widely needed throughout systems. Think physical store locations and such.

Services can either store it in their own database and still get an update stream, or services can read the complete set on boot and keep it in memory. It can avoid the need for RPC/REST calls so centralized services that keep track off this kind of data, preventing cascading failures if those fail.

[1] https://en.wikipedia.org/wiki/Master_data

It works great with the right kind of data.

What are the wrong kinds of data? It would be nice to know before I fall into a trap.

Franz is a fairly obvious name for a Kafka related tool/library so hopefully it won't get confused with https://github.com/twmb/franz-go

There's even the same in Swift (just the library as in Go):

https://github.com/kellanburket/franz

That sounds even closer to home in that case!

It is also the name of this multi-provider messaging app: https://meetfranz.com/

Yeah this is the one that's the most well known imo.

I saw defn (Clojure) in the URL and thought about https://franz.com/ immediately.

I would personally go for something like "Samsa".

Looks great - beats the dependable but obscure CLI tools I’ve been using. Are you considering selling the app? If not, what about open sourcing it?

Thanks! Yes, I plan to eventually sell it. I’m debating with myself between doing a shareware model where it’s perpetually free with the occasional nudges to buy a license or a more straight forward trial plus $30-50 license. Probably the latter would be more sustainable.

Perpetual nag, my employer will probably never purchase. Time limited trail? We may purchase 20-30 keys.

FWIW I don't think the macOS + Kafka (professional) userbase needs the shareware model. (I'm in neither groups)

Very cool project! I wanted to listen to your video, but the microphone noise was too much.

I hope you do a walk through of the code for this.

Sorry about that! My audio recording skills are subpar. Yes, likely my next video will be a walkthrough of some of this code. I’ll do my best with the audio on that one.

You could try running it through Auphonic's free tier https://auphonic.com/pricing

yeah just grab a cheap external microphone, it will do the trick! Keep up the awesome work.

Will this support RedPanda out of the box?

Most likely it will.

At the same time, RedPanda provides a native command-line tool named "rpk" which works on macOS as well and which can manage topics and consumer groups. Which makes me think that not only would RedPanda users select this, but perhaps it's also a better generic CLI to talk to Kafka?

I have yet to try RedPanda, but if it is API compatible with Kafka then it will work out of the box.

Love seeing folks create new native macOS apps. Thanks for this!

Can you manage ACLs through this? Anyway looks sweet, I’m going to try it on Monday at work. This is definitely something I’d pay for.

Thanks! I’ll be adding acl support in the next few weeks.

Seems like a neat little tool. I've always found Confluent Platform to be slow, clunky and overkill. Any plans for Windows?

Yes, if this gains some traction then I’ll probably do a Windows version down the line.

It would been great if this was pure swift + C that would have made this much more publicly usable.

I get wanted to do cool things (and experiment in spare time) and admire the work but part does feel a missed opportunity too.

Either way not my say so at the end this is your work and you should be proud of it :)

> It would been great if this was pure swift + C that would have made this much more publicly usable.

Can you say more? I don’t follow what you mean by publicly usable here.

One problem I foresee is all my Kafka was behind layers of firewalls or restricted networks only accessible through jump hosts.

In other words, to make this easier it might be a good idea to have a builtin ssh/(generic CMD) kind of way of opening streams.

what kind of authentication mechanisms does it work with? We have an oauth and ldap backed cluster. This tool would be a nice drop in tool for diagnosing issues.

Will try next week at work

Another authentication option I'd be interested in is AWS MSK with IAM authentication.

I'd like to second this request. All of our Kafka clusters are in MSK with IAM auth. I've been desperately looking for another management tool to replace Conduktor (conduktor.io).

Noted! I'll probably work on this in the next month or so, along with a couple other suggestions I've received from folks.

Currently, only SASL is supported since that's what's baked into the protocol, but I can look into supporting LDAP, too (pointers appreciated).

Wow! Just in time, can't wait to use this during work.