This article appears to be comparing MQTT and Kafka + Schema Registry. Using Schema Registry is not required to use Kafka, so OP overcomplicated their own set up for this comparison. There's no argument that Schema Registry is valuable, but it's not something that MQTT seems to provide out of the box, so the comparison seems flawed.
I'd be interested in a comparison that is actually apples-to-apples instead of introducing complexity with Schema Registry.
Comparing mqtt to kafka is already apples vs oranges. Adding the schema registry to this is like throwing a pitaya into the mix.
Edit: after reading the article a couple of times it’s clear this isn’t a comparison. “Vs” in the title is the problem. The first impression would have been better if the title was something like ”Kafka and MQTT“.
To be honest, as kafka and mqtt often reside next to each other, they complement each other. A better use case would have been an app combining both: show when should mqtt hand over to kafka.
Can confirm, I work with several workloads where MQTT is used to aggregate IoT data close to the source, which is then pushed into the cloud which uses Kafka.
The author is also incorrect about message keys - these are optional and you would only use them for strict ordering when using > 1 partition, or log compaction where "latest is greatest" is good enough.
The article is pretty biased by comparing the complexity a schema free scenario (MQTT) to Kafka with Schema.
However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster.
And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server for 10% of the operating cost.
Kafka is a monster of complexity notoriously hard to operate (Hello ZooKeeper) and to understand properly (Hello ordering, persistency and partitions).
If all you need is a simple stupid publish/subscribe broker with topics/auth management, do a favour to yourself, stay away from it.
> However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train on event sourcing and a recipe for disaster.
While calling this out on a message board comment section is going to be well-received, asking "do we need this" while working at the company with said architect/tech lead is not well-received.
How many of us get paid to work jobs where we're basically told "shut up, this is what we're doing/using, go with it"?
And go for mqtt instead? Not a smart choice. Look, it’s going to work for a few hundred, few thousand topics. But as soon as you need resilience, replication, or you outgrow that one broker… good luck. Mqtt is awful to scale horizontally.
This comment seems backwards to me. If you're funnelling incoming messages in to hundreds of topics (or less) Kafka is a great "fat pipe" if you need millions (or tens of millions) of topics for IoT devices, MQTT is much more designed for that usecase
Disclosure: I'm biased - I've worked on the MQTT spec and I'm the lead for Eclipse Amlen
You are absolutely right. That's what I said in my sibling comment so I'm not sure what's backwards. However, if your millions of little topics don't fit on a single machine - what do you do? You need a fat pipe. Hence you put your little topics into the fat pipe, send over the fat pipe to other mqtt brokers that need to disseminate egress.
Example - you have 1 topic with 1 producer and 20M consumers. Each consumer is a tcp connection. Say that you can do C1M happily, you still need 20 brokers to serve egress for all your connected clients. Now imagine that you have 100 brokers, 100M connected clients and your connections are randomly distributed over your brokers. You don't want to route every message to every broker because. So you need a fat pipe and some middle man that knows which brokers a message must be routed to because there are currently consumers connected to those brokers waiting subscribed to topics and waiting for messages. As someone who works on MQTT, you for sure understand the problem.
I have never heard of Eclipse Amlen. However, I am working with MQTT and Kafka since 2012 and have seen a nation-wide successful MQTT rollouts where MQTT and Kafka worked in tandem to solve exactly the problem you are talking about - millions of little concurrent connections distributed over a large fleet of devices for sub-100ms round trip.
I agree there are cases where Kafka and MQTT are often used together. If you have lots of MQTT clients producing messages fanned-in to a small number of backend apps (or consuming a small number of wide fan out messages) people often combine Kafka as a fat pipe behind MQTT brokers (though there are alternatives, consuming messages from the brokers using e.g. MQTTv5 shared subs)
In more complicated situations (e.g. "outgoing" persistent messages buffered for individual clients (i.e. each client has a "message inbox"), Kafka is less obviously useful (it's an anti-pattern to try and random-seek the messages from Kafka topics as clients connect). In this kind of pattern, the main architecture pattern I see are clients partitioned across (highly available) MQTT brokers. If the messages come from MQTT clients directly to other clients (e.g. instant messaging (Facebook Messenger uses MQTT), having these broker in a cluster sharing a topic tree is very useful.
If the outgoing messages don't get buffered whilst the clients are off-line (e.g. because these are responses to client requests) then you don't need each client to be routed to a "home" broker that buffers the messages - it can connect anywhere.
It all depends on the shape of the message flow you're designing the system to support (but to say MQTT is a toy for small numbers of topics as it seemed to me that you argued up-thread seems misguided).
> So you need a fat pipe and some middle man [...] waiting subscribed to topics and waiting for messages.
Your use case is not Mr everybody use case nor the one presented in the article. Most usages of Kafka I have encountered in the wild is for notification delivery or telemetry report and of the order of few ~1000 msgs/s.
You do not need a fully distributed ordered log system for that. MQTT does the job for a fraction of the complexity and operational cost.
> . Say that you can do C1M happily, you still need 20 brokers to serve egress for all your connected clients. Now imagine that you have 100 brokers,
Even at these scales, you can find some commercial MQTT brokers going over 20M msg/sec nowadays.
With OSS solutions, you could get your way there with some HAProxy + your favorite MQTT broker behind DNS load. balancing as long as you do not requires HA, scale only should not be the issue.
It would even play pretty nicely with anycast if you want to place your brokers at edge close to your customers and do some proper partitioning.
That is currently pretty much the case presented in the article. They just advertise telemetry report (very likely not HA) injected by a time series database.
Once again, if what you need is fully ordered distributed commit log for a complex scenario of event sourcing: Go for it, go for Kafka, it has been designed for that.
But it is just not the case of most Kafka instances I see deployed in the wild, these ones are generally the result of quick Google-search driven engineering.
Soooo right around when you have actual business with actual customers and use cases and are in best moment to re-architecture your app properly instead of throwing infrastructure at wall and seeing what sticks
Then just shove MQTT topics into kafka and slowly migrate stuff over.
I worked on a smart home platform where we did exactly that. But we moved from RabbitMQ to Kafka. This was a Spring Boot microservice architecture. We moved all of it without changing the service consume/publish semantics. The added benefit was that to facilitate the move, we wrote full integration tests on real containers and we have replaced RabbitMQ with Kafka under those tests. The system handled much better under load and the company had full integration tests. The migration took 2 weeks with 8 weeks writing tests for a team of 3.
Those are two different technologies. Amqp is all about routing and queues. Kafka is a distributed log, it is not a queue. There’s a significant difference between those two.
Kafka: every consumer for a partition within a consumer group will see a message at least once. A queue: it’s possible that a partition has multiple consumers and only one consumer sees a particular message.
Kafka is relatively small to medium number of large volume topics. Topics can be larger the a machine due to partitioning. Strict ordering per partition based on message arrival time. Queues (RabbitMQ or anything amqp) are relatively large number or small to medium volume topics. Ordering is an option, a topic must fit within a machine.
Thos are orthogonal concepts. They can live next to each other. My first choice is always Kafka because: persistence, replication, scalability. Works fine as a single broker, can always scale horizontally, zookeeper is not that scary, especially with a good operator.
If you think that Kafka is difficult to run, wait until you need replication in RabbitMQ. Good luck with leader election in your application layer. No fun.
Kafka was designed for places with a huge data/log volume (GB/sec or more).
From what I understand it's generally The Thing To Use when you have not very many (ie: thousands) high-volume sources that have good links. You push stuff into the logs and you can take your time chewing through the log.
MQTT was designed for shitloads (hundreds of thousands+) of small devices on bad links connecting to the mothership.
Neither of them really are a message queue. I mean, they queue messages, but they have different goals than, say, ActiveMQ or RabbitMQ (which both can be used as a backend for MQTT).
Using Kafka for a message queue is overkill. It's more likely that MQTT fits your bill.
> When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed?
Lots to unpack here. The simple answer is: you use Kafka when you need to have the data in order in arrival written to disk but you don't need ad-hoc query. Messages must be persisted, processed at least once by someone, in order. It's common to have Kafka in front of a database as a sort of buffer: when data is in Kafka, it has been acknowledged as seen and will be processed later.
Now, why would you store it in Kafka rather than in a table with timestamps. I said in my previous comment that messages are persisted based on the arrival time on the broker. This is somewhat inaccurate. Kafka is not related in a message arrival time. It's interested in message arrival. All messages arriving on a partition are process in order of arrival. Messages are stored on disk as byes, each message is written to a segment file, a partition will often have more than one segment file. Segments are sequential. A message is written at a byte offset within a segment and the format is something like: {metadata, length, message body}, the protocol is documented here: https://kafka.apache.org/protocol. Each segment file has an index file. The index file maps the partition message offset to a byte offset within a segment file. To read a message, the broker needs to know the message sequence you want. The broker will identify the segment by looking at (AFAIR) the index file name which contains the first message offset within its respective segment. It will read the index, find the byte offset of the message, check the message length, read the message, and return it.
However, as you can imagine, this is a pretty involved scenario because it implies a bunch of random reads from disk. The point of Kafka is sequential reads over complete partitions or segments. So Kafka shines when you need to read large sequences from a partition. It will identify a segment, memory map it and do a sequential read over a large file (usually a segment is something like 1GB).
This is different than inserting into an SQL table with timestamps. First of all - you will have duplicated timestamps. The only way to avoid duplication is auto-incremental indexes. These will be similar to the message offset. However, a large number of writers will inevitably lead to a congestion on your auto-incremented sequences. Next, your table will have indexes: as you want ACID, those indexes will have to be updated before data is considered stored (C). Next, you have the data stored but it's an SQL database, so your data is on one node. This is different than Kafka minimum replication factor where the data is considered as stored when at least N replicas confirm it. As for reads: so your client application would store it's own last processed sequence number from your auto-incremented index, and this would work the same as the consumer offset in Kafka. But it you have hundreds of GBs of data on your server, you'll be hitting disk pretty often for index scans, or your indexes will outgrow your RAM. Sure, you can scale to a bigger box (costly) or you go for a DRDBMS (YBDB or Cockroach or similar) but due to how your tables are distributed in those systems, you will hit a latency cliff. Reads are usually better than writes, writes can be atrocious, DRDBMS large batch writes can be 500x+ slower than single node RDBMS.
> so you can replay them/fetch them if needed?
If you need ad-hoc queries, that's the best way to go. But for volumes of data larger than your SQL server, I'd consider Kafka in front and SQL to store only what needs to be queried.
> Why do you need a sharded Kafka cluster?
Three reasons:
1. Write performance. I wrote higher that messages are written in order into a sequence file. I have personally worked with clusters ingesting 400Mbps per partition on multiple partitions on a single node. Basically, Kafka can saturate your bandwidth and if you have SSDs, everything what's ingested, will be persisted because Kafka uses zero-copy and memory mapped files. So it's basically like writing to RAM. If you need to ingest more data than a single node can handle, you need more machines.
2. Read performance: data within a topic partition is written in sequence and will be read in sequence. Your typical scenario if using some derivative of a key to establish the partition number. By default kafka uses murmur2 hash. What this basically means. Imagine that you have data for email addresses. The email address is the key. All data for a given email address ends up on the same partition. Thus your consumer will always read from one partition for a given email address. The more partitions you have, the higher is the number of consumers. Of course one there will be multiple email addresses within a single partition because their hash maps to the same partition number. The higher the number of partitions, the higher the granularity of that distribution. However, it's easy to hit a hotspot for a key if you have a lot of data for a hot key.
3. I would argue, the most important one: replication. If you have 3 brokers and your topic has a replication factor of 3, you have one copy of data per physical machine, all data consistent across brokers. That is - each partition and every segment exists in three copies, one copy per physical broker. There will be multiple partitions for the same topic on the same broker and each will be replicated to other brokers.
Taking a broker out does not impact your operations. You can still read and write, you will have under-replicated partitions but Kafka will automatically recover once the missing broker comes back. If a broker falls out, and that broker was a leader for a partition, Kafka will elect a new leader from one of the remaining brokers and your consumers and producers will automatically continue writing like nothing happened.
> Most businesses are going to have Redis, SQL, and probably RabbitMQ.
> Where/why add Kafka to that stack?
Kafka is an all-purpose strictly-ordered-within-partition buffer of data with automated replication and leader election. Multiple consumers can read from the same topic partitions at the same time. Each one has its own offset and can consume at its own pace. Each consumer can do different things with those messages at the same time. Those consumers can be located at different org levels. Kafka is great at moving large volumes of data within the organization and facilitating other processes, for example ETL.
Man, I sound like I'm some Confluent salesman. I'm not affiliated. I work for Klarrio, we build streaming systems.
Thank you for this. I'm currently involved in the design of a large-ish IoT backend having only worked on smaller volume systems using "traditional" brokers (MQTT, RabbitMQ). Your post helped me fit Kafka into my mental image.
What do you think about an incremental development where one starts with direct MQTT subscribers and adds Kafka only once the volume goes up?
Without knowing much of your exact requirements, I would aim for the following: devices communicate over MQTT, support one protocol, do it well. The problem with MQTT is scaling the broker. It all depends on what is behind "large-ish". Hundreds of thousands of devices? Millions? Dozens of millions? If you can fit all connections on one broker, it's easy, any available solution will handle this. Going past one broker is where the problem starts. Imagine the following scenario:
You have 10 brokers, each broker handles 1M connections. You have a TopicA subscriber on broker 9 and 10, someone publishes a message to TopicA on broker 2. Your broker 2 needs to forward the message to brokers 9 and 10 but no others because if you do this for every message, you waste a ton of compute. So you need a method to distribute your connection table: which brokers currently have connections with subscriptions to TopicA. This is reasonably easy to embed in the broker for a relatively small number of brokers but if you need hundreds of brokers, your global connection table will not fit on each broker. Scale this up - millions of topics and millions of connected clients. You need a way to offload the connection table somewhere else, find a reliable method to distribute changes to your connection table. This is where we have successfully used Kafka - distributing the connection table and state changes.
Got it - I think for the first 1-2 years one broker instance should be plenty. So the decision to up the complexity can probably be postponed to the point where it becomes clear that we hit 10M+/lots of brokers will be required.
Definitely. Focus on solving the business problem first, grow the business. When you have the need to make a switch, you will find a way forward by rebuilding your your MQTT infrastructure to fit your scale. If you ever need to.
Yeah - business is smart home renewable energy stuff, not lightbulbs. If they don't go international real fast the small oldschool broker cluster solution will probably be plenty for the next 5 years.
If you can live with it (low traffic, small data set) you totally should. Where ‘low’ and ‘small’ are based on what a modern server can do, so they can go up quite a bit. And with shards and/or replicas you can get N times the throughput in an easy way
> When should you use Kafka instead of storing rows in SQL with a timestamp so you can replay them/fetch them if needed?
When the use case warrants it and when downstream consumers need to be notified of a change in the data or its state. That is, in reactive architectures.
A data streaming platform (of which Kafka is one example) will push the data, whereas a database (relational or not) requires to be explicitly polled (queried)[1]. A poll for new data is inefficient, is computationally expensive (a waste of CPU cycles when the data has not changed) and is prone to create creating delays in the data processing (when the polling interval is too long). Data streaming platforms will deliver the changes in near realtime.
Also, Kafka, being a distributed append only log, keeps an entire temporal history of all data changes. That is, if an event or an object has undergone a series of state transitions, such as create -> update 1 -> update 2 -> update 3 -> delete, it will be fully available in a topic as five separate events. If a consumer is only interested in the latest state change, a series of events can be compacted into an equivalent of a database table (naturally, it is called a K-table in Kafka) and the consumer can use the K-table instead of having to process the entire event stream[2]. K-tables and event streams can be flipped over with no extra effort.
Kafka is, effectively, your databases's own transaction log sans the query engine.
> Why do you need a sharded Kafka cluster?
Kafka cluster are not sharded. At least not in the database sense, as it is a cluster. A consumer sees one living thing[3] – the cluster, and the cluster can be dynamically resized without the consumer(s) knowing or noticing it. The cluster will then rebalance itself.
> Where/why add Kafka to that stack?
Kafka is typically used to get the data out of data sources and deliver a temporal series of changes to interested consumers. If the data needs to queried ad-hoc or explicitly, Kafka is then fronted by one or many (usually, domain specific) event stores. An event store can be a database (a RDBMS or a document one), a document search engine (e.g. Elastic Search or a vector database) or anything else that has a query engine in it. It can be a cache as well when either the speed of retrieval or affinity to the data are important.
[1] It is possible to mimic the Kafka temporal history via the use of triggers that would inject a copy of the row being changed into an «audit» table before allowing a update to occur, but, with time and in high data volume environments, the table size will balloon and will drag the database down.
[2] Most databases now have change event streams that abstract the database transaction log away and deliver raw row changes as a temporal history of events, which is a functional equivalent of data/event streaming (using Kafka or similar) sans the scalability and the distributed processing – the onus to implement either or both is entirely on the consumer and that part is hard. Therefore, a database change event stream is oftentimes hooked directly into a Kafka cluster.
[3] Not entire true in most implementations, but it is rabbit hole for another time to go down into.
This is one of the major differences from MQTT. MQTT broker pushes to the subscriber when the latter is connected, or stores for redelivery if subscriber is offline (qos 1 and 2) but the redelivery will also be a real push on reconnect.
You are absolutely right, but not every project needs reliable messaging.
Many projects can be just fine with ephemeral messaging, they might already have some kind of managed scalable database and failures can often be handled on the client side.
Often we insert messaging into an HTTP system, or replace an HTTP system. In those cases, the client is waiting for an acknowledgement from the system that sits on the other side of the queue.
What happens in HTTP is you send a request to an HTTP server, but it combusts before you get a responce? Connection breaks or you get "request timed out" and user can click the button again. Or you can retry in the client code.
For many joke apps it is not always worthwhile to double complexity to move from a 98% system to a 100% system.
NSQ is a message broker that's intended to run in a distributed fashion, i.e. one broker per machine which is producing messages. Consumers then have to discover and connect to all brokers that carry messages on a given topic. NSQ persists messages to disk, but it does not have replication, so if one of the machines with the brokers dies (as in dead disk), all queued messages will be lost.
We use NSQ extensively and its a great fit for us. It's fast, lightweight, very easy to understand and work with, has great HTTP APIs for controlling the queues and publishing messages, etc. What it doesn't have is replication, so if a message must not be lost ever I'd reach for a distributed log, probably looking at Redpanda before Kafka, and possibly building my own on top of ScyllaDB since we're already running that.
I haven’t used it so I’m not qualified to answer, sorry. I can only tell why I never used it: as far as I understand it’s possible to have data loss with nsq and there’s no replication.
Ha, but do you want AMQP 0.9.1 or 1.0? Completely different protocols. And do you need a router? A broker? An L7 load balancer? So maybe you need Apache ActiveMQ next to your RabbitMQ with AMQP 1.0 plugin? Or Qpid? Or all three of them?
What I liked about a mix of three of those was that I could connect Azure Event Hub with RabbitMQ and ActiveMQ, they would all speak to each other via Qpid.
>Kafka is a monster of complexity notoriously hard to operate (Hello ZooKeeper) and to understand properly (Hello ordering, persistency and partitions).
100% this.
Even using managed Kafka is a pain for most use cases. We replaced managed Kafka with a simple postgresql db using skip locked as a queue mechanism and the dev teams productivity tripled and our total cost of ownership decreased dramatically.
Don’t think twice, think 10 times if you really need Kafka
Kafka no longer requires zookeeper. If you need true master-master high availability from a datastore - which anyone who bothers with a load balancer for their application should demand, what's the point in running your application in a HA configuration if your datastore is a single point of failure - then to the best of my knowledge Kafka is still the least bad option available. It's not the easiest thing to operate, but I'll take it over Galera or Greenplum any day.
I see Kafka deployed for things which have perhaps a few thousand messages per day. It's like "did you accidentally mis-specify by six orders of magnitude here?"
>However his points still remains: Most of the usage of Kafka I have seen in production are the result of a random Architect/Techlead who tried follow the hype train
Don't look now but this is how people end up with k8s as well. "We need Kubernetes because we need containers." Google et al convinced people it's the only way to run containers in prod.
I really have to disagree -- for what I've used it for kafka's basically been reliable and amazingly simple to manage.
The use case is, I think in the grand scheme of things pretty simple -- it's an "on-prem" infrastructure with some mixture of old and new servers and mixture of SSD and terrible old 3.5in rotational media. None of it is "cloud" -- just kafka clusters deployed with puppet with fluent and kafka rest proxy feeding the kafka and logstash or vector reading from it, but ... it just works. We've had one incident in the past 4 years, and that was because the network decided to go super asymmetric.
Anyhow, I've got lots of problems, but "running kafka" isn't one of them.
> And in 90% of the case, that could have been replaced by a trivial lightweight mosquito (MQTT) server for 10% of the operating cost.
What about ZeroMQ and if one also needs to temporarily store the queued data at least until it's delivered?
We use MQTT now, but with EMQX as a broker instead of Mosquitto. It has a HTTP API for managing users and ACLs which was easier to integrate than the equivalent Mosquitto MQTT API.
One missing criterion is client complexity. MQTT is built to work well with very little resources on the client. Kafka, on the other hand, requires you to do things you just don't want on a small embedded device -- like opening multiple connections to multiple hosts. Kafka is also just a transport for messages while MQTT is much larger part of the stack and takes care of transporting individual values. Which means you need less other code on your super restricted device.
That said, I don't understand all the complaining directed at Kafka in this thread. Kafka is a fantastic tool that provides unique properties and guarantees. As a tech lead/architect I love to have a good selection of tools for different situations. Kafka is very reliable tool that fils an important role of when creating distributed systems and is particularly nice because it is easy to reason about. The negative opinions I heard in the past are typically from people who try to use it for something that it is not well suited for (like efficient transfer of large volumes of data) or because they misunderstood how to use its guarantees to construct larger systems.
At one place I met a team who was completely lost with their overloaded Kafka instance and requested to get external help to "further scale and tune" it.
I just touched the piece of code on producer and on consumer to publish data in large files to S3 rather than push it all through Kafka. Instead, send a simple message to Kafka with the metadata and location of the payload in S3. And then the client to download it from the bucket. They were happy puppies in no time.
Regarding your last point, how did you handle deletion from S3? Did you not need to worry about atomic consumption of the metadata and data? I suppose you could have some kind of background gc task..
I think you are overcomplicating the problem for no reason.
You build your system from simple guarantees:
* message to Kafka is sent after the payload has been published to S3. This means if you have received message on Kafka, the payload is there, no need to worry about it -- you guarantee it because of order of publishing operations.
* the object on S3 is immutable. This means it does not matter when you consume it, it stays the same.
* the message on Kafka is immutable. This means it does not matter when you consume it, it stays the same.
When the client reads the message off of Kafka topic, it just downloads the additional payload from S3. The payload is guaranteed to be there and exactly the same content as published. Once the message is fully processed, it commits this to Kafka topic and that's done. If the processing fails, the processing will be retried later by this or another node. The payload is still there until somebody decides to delete it.
Deletion can be done in many different ways. You could have metadata for all those objects (Kafka topic is your metadata database!) and see what is the oldest timestamp on the offsets on all partitions still not committed. Then you delete from S3 all objects that are older that that. This requires that you publish to Kafka in the same order as you publish to S3 (within each partition).
Yes I understood your proposed solution (and have no particular issues with it). I was specifically asking how you went about deleting things (i.e. the last paragraph of your response). Did you actually implement it that way in the end?
The vast majority of "Kafka problems" (or rather "Kafka Streams problems") we have with a managed solution at work are due to not fully understanding how it works and how to set it up. There's so much stuff you can configure and so much potential to misuse it. Typical problems are wrong configuration for acks, not understanding durability guarantees, not understanding exactly-once semantics, not naming hidden topics in Kafka Streams, not using schema registry serializers for hidden topics, choosing the wrong partitioning scheme (and wanting to change it later), using Kafka clients with different hash functions, using wrong schema compatibility mode, etc. etc.
From a technology perspective it's been rock solid for years in my experience.
Where issues crept in it was always due to people not understanding the architecture and patterns you need to use e.g. anti-patterns like splitting batches into multiple messages, "everything must be stored in Kafka" thinking, not understanding how offset commits work, not understanding when to use keys or the effects of partitioning, resetting offsets on a live topic, aggressive retention policies etc.
One issue I’ve encountered is over-partitioning to handle a spike in traffic.
I.e. an event occurs which causes an order of magnitude more messages than usual to be produced for a couple of hours, and because ingest and processing flows are out of whack, a backlog forms. Management wants things back in sync ASAP, and so green lights increasing the partition count on the topic, usually doubling it.
In an event driven architecture that is fairly well tuned for normal traffic this can have the same downstream effect, and those topics up their partition counts as well in response.
Once anomalous traffic subsides, teams go to turn down the now over-partitioned topics only to learn that that was a one way operation and now they’re stuck with that many partitions, and the associated cost overhead.
Also if I see another team try to implement “retries” or delayed processing on messages by doing some weird multi-topic trickery I’m going to lose my mind. Kafka is a message queue, not a job queue, and not nearly enough engineers seem to grok that.
If you’re on AWS I’ve had zero issues with their managed Kafka offering (MSK). I’m sure they did lots behind the scenes, but it was really one of our most rock-solid pieces of infrastructure.
If I had a need for Kafka in my current role, I’d probably give Confluent and Red Panda offerings a shot.
> In the places I've worked that use Kafka, it's 100% always a source of issues and operational headaches.
Compared to what?
I have the opposite experience. For example, ingesting large amounts of log data. Kafka could handle an order of magnitude more events compared to Elasticsearch. Even if the data ultimately ended up in ES, being able to ingest with Kafka improved things considerably. We ended up getting an out of the box solution that does just that (Humio, now known as LogScale).
Similar experience when replacing RabbitMQ with Kafka. None "just works" and there's always growing pains in high throughput applications, but that comes with the territory.
Is Kafka the source of headaches, or is it Zookeeper? Usually it's Zookeeper for me (although, again, Zookeeper has difficult problems to solve, which is why software packages use ZK in the first place).
I hear from everyone using Kafka in production that it is hell unless you use Confluent.
I gave a try to NATS JetStreams but I havn't been convinced by the performances of the Python client, nor the JavaScript one. I don't have extreme data, I just need descent performances.
I'm thinking about giving a try to RabbitMQ streams. I have been very happy with RabbitMQ, the MQTT plugin isn't fully working (the big one is that retained messages are not sent to wildcard subscribers), but it should work with AMQP.
I've run self-managed, sorta managed (MSK), fully managed (Confluent Cloud), and somewhat managed (Strimzi).
It is complex, yes, but it solves a very complicated problem. The issue tends to arise when people use it when simpler alternatives exist for their problem.
I've had an excellent experience using the Rust NATS client!
I pump time series data through NATS running on a Raspberry Pi, which is part of a 3D printer monitoring and event/automation system. I also use NATS as an MQTT broker, for compatibility with other software in the 3D printer ecosystem.
FWIW I also have lots of experience running large Kafka and Rabbitmq fleets. The choice between these technologies depends on what you're optimizing for.
We went the opposite route. Kafka has been much better. Up to a certain volume, both solved the problem. When RabbitMQ required too much tuning, a decision was made to go to Kafka, and it's been stellar.
Both are pretty good, but understand that there are too many variables involved and you can't really escape production hell indefinitely, regardless of what you pick. What changes is when you are going to see the flames, and what is going to spark them.
Have you tried Pulsar or Redpanda? Both seem mature enough and provide decent performance to probably meet your needs. What I hear is that Redpanda is a lot easier to manage than Kafka.
Redpanda Community Edition is licensed with the Redpanda Business Source License (BSL). The core features are free and source-available.
This license was inspired by MariaDB and CockroachDB and "for 99.999% of users, restrictions will not apply". The big restriction is that users "cannot provide Redpanda as a commercial streaming or queuing service to others" which is primarily to deter large cloud vendors from taking our work and impacting our ability to operate as a business.
Here are some links for more details on how we came to this decision and how we license all our products and features.
The blog post series seems to bury the lede -- it isn't until part 3 [1] that we get to the insight that MQTT and Kafka solve different problems and therefore can have complementary roles in the same system.
We use this architecture for IoT: MQTT for edge because it's standard and super good enough, and Kafka because it turns out we want to do more than one thing with the data as it streams in so not using Kafka would end up being more complicated.
Here's a key insight though: for IoT you don't want to use an actual MQTT broker, like Mosquitto or HiveMQ. If you do, it's hard to avoid data loss. Server side you have something that subscribes to those MQTT topics and pushes the data into Kafka. What do you do when that thing needs to be restarted? Ok, you can use persistent sessions in your MQTT broker. But how much memory does your MQTT broker need? What if your MQTT broker crashes? Oops, now your MQTT broker needs its own persistent database to keep track of all those messages in limbo.
What you want is an MQTT gateway -- something that looks like an MQTT broker to the devices, but the server side does something different with received messages. When it gets the MQTT PUBLISH command, it sends the message to Kafka, waits for the ack from Kafka, and only then sends PUBACK back. Presto, the MQTT thing is now stateless and horizontally scalable. Your clients just need some retry logic.
Maybe Kafka's mqtt-proxy does this, I don't know. I don't think it's mentioned in part 3. But it's a key property of such a system. I'm guessing Amazon's IoT gateway does this, because once you've thought about it hard enough it becomes obvious this is how it needs to work.
Interesting, can you explain how that proxy would avoid the problems of the something on the server side you mentioned? I would have guessed it will have all of the same problems you mentioned, with requiring MQTT persistence etc if the proxy goes down/has to be restarted?
If the proxy is stateless, messages from client devices aren’t confirmed until Kafka ACKS them, so messages either reach Kafka, or they don’t. If the proxy goes down, nothing is lost, because the client hasn’t been told that they message has been handed off completely, so they simply retry. In the stateful/broker case, you incur extra bookkeeping because you told clients their messages were delivered, when really, they’re just buffered (with all the extra overhead that entails). The stateful approach also hides system back pressure from clients, so in the event of network/storage/service degradation/failure, rather than backing off producing messages, or applying some other kind of application specific decision; clients keep producing messages as if nothing is wrong, because the stateful broker is buffering them locally. Storage and resource reqs for the broker continue to grow, until they either suddenly die, or stop accepting messages. Even worse, when downstream comes back up, the brokers are they going to dump all their messages and possibly re-overload recovering systems?
Thx, now I get it - so the difference between the something and the gateway is that something would be an actual MQTT subscriber, while gateway is actually just a facade implementing the MQTT protocol backed by Kafka.
Sounds neat, but I suspect conforming to the MQTT specs but not actually being a 'full' broker might be easier said than done... Basically it will have to simulate MQTT behaviour while actually living by the Kafka rules.
there’s a lot of “Kafka causes so many issues!” comments here.
I think it gets a bad rap because it gets introduced to orgs without the org having the requisite level of understanding.
If your whole org is just on like a standard OLTP/OLAP setup, then suddenly there’s a Kafka queue, there’s going to he a serious learning curve and bumps along the way.
If you’re incorrectly putting async event brokers as the datastore where you should be putting a synchronous DB and then streaming from the DB to kafka with an outbox pattern, you’re going to have a bad time.
If you’re not modeling your queue depth and throughput you’re going to have a bad time.
If you’re not modeling your concurrency scenarios and synchronization, you’re going to have a bad time.
Since nobody else mentioned it: Often you'll want to have multiple consumers of a given topic for load-balancing/failover - that's on the consumer side. The support for this in the MQTT standard is poorly defined, and popular libraries handle it poorly to the point that I can't recommend trying it. Message queues right-and-proper load-balance consumers by locking the entire topic once per delivery; Kafka does it by assigning "partitions" of a topic to different consumers, and thus you have faster, lock-free delivery (Kafka is essentially a "hot-rodded" message queue, just like hot-rodding a car by removing certain "proper" guard rails).
I've been combing the interwebs and following countless tutorials for different types of IoT data gather solutions using various messaging services and brokers for an scale weighment system that I've been developing using VueJS (eventually looking migrate to Nuxt).
So far the concept is simple, a weighment scale has an RS232 (COM) port that streams data out for the tare, net, and gross weights using some kind of micro-controller architecture, Raspberry Pi's, Arduino, PyBoard etc. via some sort of messaging service.
Clients connect to a page where they can "pub/sub" to scales to see the current weight of specific scales and using a remote device. The page acts as a UI to either manage inventory, register consumed ingredients, or register completed tasks.
So far most of the development scripts and test environments I've built all seem to do basically the same thing and will gather the data that's needed but I'm curios if anyone has dealt with anything like this in the wild and if there are any caveats that I'm missing or other technologies that would be better suited for what I'm trying to achieve?
Right now Kafka or RabbitMQ seem to be my main choices for message brokers mostly because they are fairly easy to setup via Docker. If anyone has any recommendation on libraries I should look into that would be awesome! The UI is coming together nicely, I started it in React but switched to Vue3 after I fell in love with the component architecture and composition API.
Having been down this road, I strongly recommend you lock down your requirements before drowning in the sea of transport options. There are a lot of offerings out there that, from the device perspective, basically amount to a reinvention of HTTP(S). Many of these systems exist with the intent to retain the underlying TCP (really, the expensive-to-establish TLS) stream, which is not as much of a concern after HTTP 1.2 retained existing connections.
What are your latency requirements for RS232 to DOM paint? Are you storing time-series data, or just the last value? How fast does the data change, or, when do you want to be notified? Do you have circumstances that periodically disconnect your device from the Internet? It doesn't sound like you have a constrained environment i.e. a cellular connection or battery budget. It also sounds like this is a small-volume project, that you are not building the next Twitter.
If you are interested in historical data, do not overlook the value of generating a larger file on-edge, compressing it, and sending it with a boring-ass HTTP POST. If you want "fast", just set a trigger (i.e. the scale changed by more than 3 ounces in 1 second) to send a small update POST. Periodically perform a GET for config updates, and you're off to the races.
Keeping your messaging layer stone-cold boring enables you to use whatever whiz-bang backend you want, and leverage existing knowledge and tools of the HTTP world. Or just slap it in a database. If you're interested in time series, check out Timescale for Postgres.
I once had to develop something similar. The solution my company had chosen was to transmit the weights and possible errors via a Websocket connection between the UI and the scale adapter - a small Java service doing the raw socket communication.
Weirdly the commands to stop weighing and set tare, get tare, etc. would be send via RabbitMQ from the UI server to the adapter instead of using the WS connection.
I jumped onto https://mqtt.org/ to try to answer my usual use-case question about non-Kafka messaging, which is: "Do the messages get saved anywhere so you can come back and read them later?" Still not entirely sure about it.
But I did see:
This is why MQTT has 3 defined quality of service levels: 0 - at most once, 1- at least once, 2 - exactly once
I'm a big fan of advertising the impossible on the front page.
MQTT isn’t designed as a persistent log, but can fulfil some of what you might want to use one for.
Each message has a couple of flags, the first being Quality of Service, which as you quoted above determines deliver guarantees. 0 is fire and forget, with potential loss of messages. 1 will queue messages for delivery to offline clients that are subscribed to a topic (within reason, all brokers set limits on that), and 2 is often described as “exactly once”, but is in fact just a more involved dance to acknowledge messages.
The other flag is a Retain flag, which instructs the broker to associate that message with the topic it was sent to, and send it on to any newly subscribing clients when they subscribe. This is good for use cases like remote device configuration - you can send it to a topic, setting the retain flag, and then when a device comes online it’ll immediately receive new configuration.
MQTT is great as a message queue for remote devices, mostly because it’s so lightweight anything with an IP stack can integrate with it, but I’m not sure why anyone would attempt to make it a piece of core infrastructure.
eh if you read the finer print it's just a deduplication id appended to every message. blog doesn't go into detail on what happen when two client pust a message with the same it, or what happens if there is more than one failure (i.e. client fails to detect a service outage and during the service outage the message is consumed by the broker but persisting fails) but in general the usage of a at least once + a deduplication id is not something revolutionary.
With mqtt it depends on the broker, eg. Emitter.io can save them for a week etc.
Offset for a client is usually stored on the broker, so if a client reconnects, all of the messages it has missed are forwarded to it.
As mentioned in other answers, service levels have a defined meaning, which is different to the absolute theoretical meaning, and really is to do with the message aks.
Really, as the article mentions, kafka and mqtt are for different purposes, with some overlap. Kafka is all about the log, whereas mqtt is about uncertain connections. A better comparison which I've yet to see, is comparing mqtt to nats.
Lastly, kafka is much easier to administer using redpanda, which doesn't have zookeeper, combines the registry and kafka connect (see WASM runners) with the runtime, and has a very nice console for debugging.
Similarly, Emitter.io does a great job with clustering for mqtt.
I'd like to see an open source kafka-mqtt bridge that worked in both directions as they all seem to go mqtt->kafka only.
The impossibility of "exactly once" is a theorem, not an opinion.
Knowing that, the article you linked is funny. It begins with maybe a thousand filler words complaining that all of this is "poorly understood", and it's not "impossible", just "very hard". Then it gets to the meat: Yeah well, you know, it's not quite "exactly once delivery", just "exactly once semantics", and to achieve that, messages need to be idempotent so that duplicates don't matter.
> The impossibility of "exactly once" is a theorem, not an opinion.
It's quite likely that your definition of what "exactly once" means differs from the one followed by MQTT. As this issue was documented years ago, I doubt this is a relevant argument to have, unless we want to feel smart by criticising others.
That's the Kafka Streams API. "Exactly-once semantics" has a very specific meaning in the context of that particular API, which the article could probably do a better job of clarifying upfront. (Otherwise it is an excellent overview of the problem and solution provided by the Streams API.)
MQTT.org can't answer that as it's a web page for for a protocol. I've worked on platforms that do have a historian feature but it will vary from broker to broker.
(disclosure, I work on Eclipse Amlen and it does not - but people often rig it up to a subscriber that funnels (some/all) messages into databases
many people seem to not have clarity on what a distributed log is and in which architecture its useful and in which not.
if you are abusing a distributed log as a message queue, you are most of the time creating a mess.
I designed a facility IoT system with AWS products: IoT Core (MQTT broker), SiteWise (analytic dashboard), S3 and Lambda. I found the AWS offerings to have everything I needed in one place with a low cost. Added benefits were being able support NIST cybersecurity requirements (CMMC v2, Level 2) in addition to the IoT system.
I was a fan of NATS and Kafka in the past, but AWS tooling makes IoT relatively easy.
I have done a lot of work with IoT-ish data, e.g. sending location and telemetry data from vehicles and remote sensors over unreliable cellular networks.
I need the ability to queue data on the device to deal with patchy connectivity and some policy for deleting messages in the queue. For example, I might throw away unsent location updates that are "stale" to save space. I might need to prioritize some messages, e.g. "lithium-ion battery pack overheating".
I may be running on embedded hardware that is too small to run Linux. The connection to the modem might be serial.
Data size and bandwidth usage can make a difference. I might get 2MB/month for $3. Bytes count if I want to send frequent updates to get more precision on the location.
I may need to get through multiple layers of network address translation, making it different to send messages back. So keeping a persistent TCP connection can help. But using TCP for one-off connections wastes bytes, so UDP can be useful if I don't care about losing packets.
I may want to encrypt or digitally sign your messages.
So, I end up doing a lot of work to handle queueing locally, but using a pretty simple message-oriented binary protocol to send to the server. The server can do whatever it wants, e.g. write it to a Kafka queue.
MQTT and Kafka are different things. MQTT is not necessarily better or worse than Kafka, vice versa.
But, sadly, people are often caught in their own loops, when there are or can be created better options.
I have created a new, free, single binary data-service platform for IoT, JoinBase: https://joinbase.io/
This single binary data-service platform has proved:
1. Kafka is not more suitable for industry or higher performance than MQTT, even from the protocol level
With carefully crafting, JoinBase has saturated one PCIE 3.0 NVME sustained write bandwidth (25 million msg/s) in single modern node. This is, in fact, can not been done by the Java-based Kafka.
2. Use MQTT and Kafka together is unnecessary.
This only makes your pipeline more complex, expensive, but much unstable and slower.
JoinBase can do arbitrary message preprocessing and auto-view(WIP). Streaming does not have to be owned through a separate monster.
3. High performance or ease of use, has nothing to do with the size of the software if the product can be properly engineered.
5MB Single binary JoinBase is enough to beat many monsters in the IoT/AIoT data pipleine:
+ sustained batch MQTT message write throughput: ~10x faster than Kafka and ~5x faster than that of one popular broker
+ basic SQL analytics: 3-4x faster than ClickHouse
+ HTTP interface concurrent queries: ~100x higher than ClickHouse
+ ...
More could be seen in our 2022 summary blog: https://joinbase.io/blog/joinbase-2023/
There are historical reasons for all of these, of course. But it could be great that we break out of own mindset loops.
Most MQTT brokers are not great at storing data. The pattern that works the best for me in solving the "edge delivery" use case is a lightweight clustered MQTT broker (e.g. VerneMQ) with a little Lua script inside to push everything into Redis Streams (of course, clustered as well) immediately. Kafka is another alternative, but it's not that lightweight, operationally, and often Redis Streams are "good enough". With an LB in front of both Verne and Redis, this setup is pretty decent for IoT data ingress.
Good article (along with parts 2 and 3). Are there key differences in secure networking constructs (TLS, mTLS, VPN, whitelisted IPs, open ports, etc.) in the options described:
Of those, MQTT should be the only thing you expose to the outside world. Secure it with regular TLS, possibly with mutual auth for your clients if you care about that.
For the others, secure them using the same tech you use to secure your database. Ideally they wouldn't be accessible from the internet at all.
I'd be interested in a comparison that is actually apples-to-apples instead of introducing complexity with Schema Registry.