+91 98726 60544 hello@mitstech.co Mon–Sat · 09:00–18:30 IST

Choosing a message broker: queue or log

Cloud By Mits Engineering Team 2 min read
Choosing a message broker: queue or log

The comparison usually presented as Kafka versus RabbitMQ is really a comparison between two data structures. A traditional broker is a queue: messages are delivered to a consumer and then removed. Kafka is a log: messages are appended, retained for a configured period, and each consumer tracks its own position independently. Almost every practical difference between them follows from that one distinction.

The queue model suits work distribution. A job needs doing once, by whichever worker is free; when it is done, it is gone. Routing can be sophisticated, per-message acknowledgement is natural, and a failed message can be retried or moved to a dead-letter queue individually. If you are distributing tasks across workers, this is the shape you want, and RabbitMQ or a managed queue such as SQS does it well with very little operational effort.

The log model suits event distribution. An event happened; several unrelated systems may care, now or later. Because messages are retained rather than consumed, a new service can be added six months on and replay history to build its own state - which is impossible with a queue, where the data is gone the moment it was delivered. That replay capability, not throughput, is the reason most teams end up wanting Kafka.

Ordering differs in a way that catches people. A queue with multiple consumers gives no ordering guarantee across them. Kafka guarantees order within a partition, which means related events must share a partition key to be processed in sequence. Getting that key wrong - partitioning by something other than the entity identifier - produces out-of-order processing that appears only under concurrency and is unpleasant to diagnose.

The operational cost is the honest differentiator for smaller teams. A managed queue is close to zero effort. Self-managed Kafka is a genuine platform with brokers, partitions, consumer group rebalancing and its own failure modes, and it wants someone who understands it. Managed Kafka services remove much of that and cost accordingly. If your requirement is fitted by a queue, choosing a log because it is more capable buys you operational burden for a capability you do not use.

Most estates that need both end up with both, and that is a reasonable outcome rather than a failure to decide - a queue for jobs and work distribution, a log for domain events that multiple services consume. What is worth avoiding is using one to imitate the other: a queue with a consumer per interested service becomes fan-out you maintain by hand, and a log used for task distribution loses the per-message acknowledgement and retry semantics that make job processing straightforward.

Need help with this? Explore our Cloud Solutions & Migration services. Learn more Back to all news

Keep reading

More on Cloud