The general debate has been that Kafka scales and RabbitMQ is flexible, but, I’ve found that RabbitMQ wins on both scale and productivity. These supposed scaling limits do not show up on the server bills, and RabbitMQ delivers better productivity on virtually every aspect of development, especially documentation. The time and place for choosing Kafka is becoming increasingly less compelling, here’s why RabbitMQ should be your default choice.
RabbitMQ has great performance and DX while offering every message pattern. Kafka theoretically is better at the extreme scale (trillions of messages), but does not support all message patterns. Kafka also has terrible documentation and the blog posts do not alleviate the problem.
Full transparency: I have no affiliation with RabbitMQ. Kafka documentation just frustrates me.
RabbitMQ is an open source AMQP broker that routes messages through exchanges to queues and pub/sub patterns. It allows you to build any and every messaging pattern with plugins for STOMP, MQTT as well as duplex communication channels.
Apache Kafka is an event streaming platform built for high‑throughput, real‑time data feeds. It features things like partitioned logs as well as various other optimizations that improve scaling.
The most obvious reason to choose RabbitMQ versus Kafka is by looking at how much more work it is to properly setup Dead Lettering. RabbitMQ makes this a simple one liner in their docs. Meanwhile, nothing is clear at all in Kafka’s docs. Even the blog post on dead lettering for Kafka provides this confusing Python snippet. What is even worse, is if you look at the reference blog post Things become even more confusing and is now trying to sell you on Kafka Connect. Your best hope is that a reasoning model can help you set up dead lettering properly, I’ve found that Grok Think and OpenAI O3 struggle with setup for Kafka frequently.
This documentation and scaffolding problem applies to virtually everything in Kafka. Lots of unclear posts and a verbose API reference... Kafka’s documentation sucks.
I cannot stress enough, you can just simply read the docs of both tools and realize how cumbersome Kafka is in comparison. Docker Compose helps alleviate the baseline setup for Kafka, but, these types of workflow environments are increasingly becoming the bare minimum to get any developer to use their product.
Trying to explore more sophisticated workflows, or, even tap into these alleged performance gains that Kafka offers over RabbitMQ requires you to look at verbose, unclear documentation.
Let’s steelman Kafka here and say that it really does outperform RabbitMQ. For most web applications, the message throughput required is well within what is needed for a single RabbitMQ instance. A single RabbitMQ queue can handle up to 50,000 messages per second, and with multiple queues, this capacity increases further. To illustrate, consider the following typical use cases and their message rates:
| Use Case | Typical Message Rate | RabbitMQ Capacity (single queue) |
|---|---|---|
| E-commerce order processing | 300 messages/sec | 50,000 messages/sec |
| Background job processing | 1,000 messages/sec | 50,000 messages/sec |
| Real-time notifications | 10,000 messages/sec | 50,000 messages/sec |
These are pretty liberal estimates. Keep in mind a cheap $7 instance on Digital Ocean can handle these throughputs. When it comes to scaling, RabbitMQ offers some awesome solutions like Quorums (via Raft) and good old Federations.
Okay, let’s say the $7 instance doesn’t cut it anymore and you need to send millions of messages a second. RabbitMQ once again delivers an overwhelmingly better experience. You can do dead-simple one script deployment workflows to scale your broker via Quorums. This Stack Overflow post showcases how simple it is to setup.
Other than that, for both Federated and Quorum queues you simply need to run new nodes in detached mode with the queue type you want to deploy. Then run rabbitmqctl queues add_member against the upstream, and you’re done. RabbitMQ makes scaling as simple as possible. Kafka on the otherhand is verbose and confusing to get the same results.
For the vast majority of use cases, RabbitMQ provides a superior combination of performance, flexibility, and ease of use that drives productivity. It’s straightforward setup and management allow developers to focus on building features rather than managing infrastructure.
One final note: If you’re looking for something completely hands off and simple, you should use AWS SQS. It’s quick and easy, but is a barebones MQ. Just be mindful of the costs, as it is not cheap at all even after moderate use. SQS gets pretty expensive fast at 40 cents per million requests!
Pangolier