Your email service sends order receipts. You scale it to three replicas, and customers start getting three receipts per order. Each replica subscribed with its own consumer group, so each received its own copy of every message. The opposite mistake is as common: resize jobs go onto a broadcast topic, a second worker joins, and every image is resized twice.
Both bugs mix up two messaging models: a message queue, where each message goes to one consumer, and publish-subscribe, where every subscriber gets its own copy. This post compares them with a diagram, then covers delivery guarantees, ordering, replay, fan-out and dead-letter queues, and how Kafka, RabbitMQ, SQS/SNS and Pub/Sub map onto each model.
Queue vs topic: who gets each message
In the point-to-point model, a producer sends a message to a queue and one consumer receives it. Consumers on a queue compete: the broker hands each message to one of them and deletes it once that consumer acknowledges it. They are replicas of one service, so adding replicas drains the queue faster. This competing consumers pattern fits work that must happen once, such as resizing an image.
In publish-subscribe, a publisher sends a message to a topic, and the topic delivers a copy to every subscription. Subscribers are different services that do not know about each other, and adding one does not change the publisher.
Most systems combine the two: a topic fans out to subscriptions, and each subscription is a queue that one service's replicas compete on. The design question is where fan-out and competition happen.
| Message queue (point-to-point) | Publish-subscribe | |
|---|---|---|
| Consumers per message | One | One per subscription |
| Consumers are | Replicas of one service | Different services |
| Scale by | Adding consumers | Adding consumers to a subscription |
| Retention | Until acknowledged or expired | Per subscription, or a time window in a log |
| Replay | No | Yes, if the broker keeps a log |
| Typical use | Jobs and commands | Events and notifications |
How Kafka, RabbitMQ, SQS/SNS and Pub/Sub map onto the models
Each broker implements both models:
| System | Fan-out (pub/sub) | Competing consumers (queue) | Ordering | Replay |
|---|---|---|---|---|
| Apache Kafka | Each consumer group reads every record | Consumers in a group split partitions; share groups split records | Per partition | Reset group offsets |
| RabbitMQ | Exchanges copy messages to bound queues | Consumers on one queue | Per queue, one consumer | Streams only |
| Amazon SNS + SQS | SNS copies to each subscribed queue | Consumers on one SQS queue | FIFO, per message group | SNS FIFO topics only |
| Google Cloud Pub/Sub | Each subscription gets every message | Clients on one subscription | Ordering keys | Seek |
Kafka. Each partition is consumed by exactly one consumer within each subscribing consumer group (design docs), so a group never has more active consumers than partitions. Kafka 4.2 (February 2026) made share groups (KIP-932, Queues for Kafka) production-ready: consumers share records from the same partitions and acknowledge each one, without the partition limit or ordering.
SNS to SQS. SNS copies each message into one SQS queue per subscribing service, in a JSON envelope unless you enable raw message delivery (AWS fan-out guide). Each service's workers compete on their own queue.
RabbitMQ. Producers publish to an exchange, and bindings decide which queues get a copy: fanout copies to every bound queue, topic matches patterns such as order.#, direct matches the key exactly, and headers matches message headers (exchange types).
Google Pub/Sub. One subscription per service; its subscriber clients share the messages.
The three-receipts bug is a subscription per replica instead of per service. Name subscriptions, queues and group IDs after the service (billing-sub, group.id=billing), never after a host or pod.
Delivery guarantees
Brokers redeliver unacknowledged messages, so the question is when your consumer acknowledges:
- At most once: acknowledge before processing. A crash loses the message, and nothing is redelivered.
- At least once: acknowledge after processing. A crash causes a redelivery, so a message can be processed twice. SQS standard queues and Pub/Sub both deliver at least once by default.
- Exactly once: each message processed once and only once, which brokers offer only within limits.
SQS FIFO queues drop a resend with the same deduplication ID within 5 minutes. Google Pub/Sub's exactly-once delivery stops redelivery after a successful acknowledgment, for pull subscriptions in one region. Kafka transactions cover reads and writes inside Kafka. None of them stops a consumer from charging a card twice if it crashes between the charge and the acknowledgment.
The practical answer is effectively-once processing: at-least-once delivery plus an idempotent consumer that records each message ID in the side effect's transaction:
BEGIN;
INSERT INTO processed_messages (consumer, message_id)
VALUES ('billing', 'm2')
ON CONFLICT DO NOTHING;
-- 0 rows: duplicate, skip the side effect
-- 1 row: apply the side effect in this transaction
COMMIT;
-- acknowledge the message only after the commit
On the producing side, the transactional outbox pattern keeps the database write and the publish consistent.
Ordering
Competing consumers break first-in, first-out: m1 goes to worker 1, m2 to worker 2, and m2 can finish first. SQS standard queues promise only best-effort ordering anyway. Strict order needs a scope, with one consumer at a time per scope:
- SQS FIFO: order holds within a message group (
MessageGroupId), and groups run in parallel. Throughput is 300 API calls per second per action without batching, 3,000 messages per second with batches of 10, and more in high-throughput mode. - Kafka: records with the same key land in the same partition, which one consumer per group reads.
- Google Pub/Sub: messages with the same ordering key, published in one region, arrive in order on a subscription with ordering enabled. Each key is capped at 1 MBps of publishing, and a redelivery also redelivers the key's later messages.
Order per account or device, never globally: a global order means one consumer and a throughput ceiling.
Retention and replay: log versus queue
A queue is a to-do list: acknowledged messages are deleted, and SQS expires unconsumed ones after 4 days by default (60 seconds to 14 days). A log is a history: Kafka keeps records for its retention period, read or not (log.retention.hours defaults to 168, seven days), and each consumer group stores only an offset, so a new group can start from the oldest record and an existing one can rewind. RabbitMQ streams are logs too, read from a start point set by x-stream-offset. Google Pub/Sub sits between: unacknowledged messages stay 7 days by default (10 minutes to 31 days), acknowledged ones optionally, and subscriptions can seek back.
Replaying a Kafka group after a bad deploy, with its consumers stopped:
kafka-consumer-groups.sh --bootstrap-server localhost:9092 \
--group billing --topic order-events \
--reset-offsets --to-datetime 2026-11-28T00:00:00.000 --execute
Replay rebuilds CQRS read models and backfills new services, so every consumer must tolerate reprocessing. The Kafka architecture diagram covers partitions and offsets in depth.
Back-pressure, visibility timeouts and dead-letter queues
Queues absorb bursts because consumers take work at their own pace, and per-channel queues in a notification system keep a slow SMS provider from holding up email and push. RabbitMQ pushes messages to consumers, so cap unacknowledged messages per consumer with a basic.qos prefetch limit.
While a consumer works, the broker hides the message from the others for a limited time, much like the leases in a job scheduler. If processing runs longer, the message reappears and a second consumer gets it too. Set the timeout above your slowest normal processing time, or extend it:
| Broker | Lease while processing | Extend with | Stop retrying after |
|---|---|---|---|
| Amazon SQS | Visibility timeout: 30 s default, max 12 h | ChangeMessageVisibility |
maxReceiveCount, then a dead-letter queue |
| Google Pub/Sub | Ack deadline: 10 s default, max 600 s | ModifyAckDeadline |
5 attempts by default (up to 100), then a dead-letter topic |
| Kafka share groups | Acquisition lock: 30 s default | RENEW acknowledgment |
5 attempts by default, then the record is archived; Kafka 4.4 can also write it to a dead-letter topic |
RabbitMQ redelivers when a consumer's channel closes before it acknowledges, and dead-letters through dead-letter exchanges.
Poison messages, ones that always fail, belong in a dead-letter queue: alert on its depth and keep a redrive procedure. In pub/sub, failures stay isolated: a broken analytics consumer grows only its own backlog, so alert on the oldest unacknowledged message per subscription.
Choosing by use case
| Use case | Model | Good fits |
|---|---|---|
| Background jobs: thumbnails, PDFs, email | Queue | SQS standard, a RabbitMQ quorum queue, a Kafka share group |
| Commands in order per account | Queue with an ordering scope | SQS FIFO grouped by account, Kafka keyed by account |
| An event several services react to | Pub/sub, one queue per subscriber | SNS to SQS, a RabbitMQ fanout exchange, Pub/Sub subscriptions, Kafka consumer groups |
| Rebuilding state or backfilling a consumer | Log | Kafka, RabbitMQ streams |
| A multi-step process across services | Commands on queues, outcomes on pub/sub | The saga pattern |
The queue and pub/sub diagram
Diagram contents
- Upload API
- resize-jobs
- Resize worker 1
- Resize worker 2
- Resize worker 3
- Order service
- order-events
- email-sub
- billing-sub
- analytics-sub
- Email service
- Billing service
- Analytics service
- send m1, m2, m3
- m1
- m2
- m3
- publish m1, m2, m3
- copy
- m1, m2, m3
How to read it:
- Two lanes, same message IDs. Top: a point-to-point queue. Bottom: publish-subscribe. Each producer sends m1 to m3.
- Top lane: the resize-jobs queue hands each message to a different worker, all replicas of one service.
- Bottom lane: the order-events topic copies every message into three subscriptions, one per service, so email, billing and analytics each receive m1, m2 and m3.
- Solid arrows are send, publish and copy steps; dashed arrows are asynchronous deliveries that consumers acknowledge.
- The Markdown block lists timeout and retention defaults.
Drawing messaging diagrams clearly
- Draw subscriptions, not only topics. Arrows from a topic straight to services hide where messages wait and who owns each backlog.
- Label edges with message names or IDs, not "events": m1, m2 and m3 show splitting versus copying.
- Show replicas when competition matters, and mark the ordering key and the dead-letter path next to their queue or topic.
The event-driven architecture diagram guide extends this notation to a full Kafka system.
Build it in Archboard
- Sign in with Google or GitHub and create a diagram.
- Add a Core Text heading for each lane.
- Top lane: add a Container from the C4 Model pack named Upload API, a Queue from the Data pack named resize-jobs and three Container nodes, Resize worker 1 to 3. Draw a solid edge "send m1, m2, m3" to the queue and dashed edges m1, m2 and m3 to the workers.
- Bottom lane: add a Container named Order service, a Topic from the Data pack named order-events, three Queue nodes (email-sub, billing-sub, analytics-sub) and three Container nodes for the email, billing and analytics services. Label the topic's edges "copy" and the dashed deliveries "m1, m2, m3".
- Color each lane with one accent, blue or teal, and add a Core Text note under the queue and the topic.
- Add a Markdown block from the Core pack with the defaults.
For a larger system, start from the event-driven architecture template; for flows already in Mermaid, the Mermaid to editable diagram tool makes a first draft.
The example uses 14 diagram nodes, within the Free plan.
FAQ
What is the difference between pub/sub and a message queue? A queue gives each message to one consumer and deletes it after the acknowledgment. Pub/sub gives every subscription its own copy, so several services can react to one event.
Is Kafka a message queue or a pub/sub system? Both. Every consumer group reads every record, which is pub/sub, and consumers in a group split the partitions like a queue. Kafka 4.2 share groups add per-record acknowledgment.
Kafka vs RabbitMQ vs SQS: which should I use? SQS for a managed queue on AWS with nothing to operate, RabbitMQ for flexible routing, and Kafka for high throughput with replayable history.
Do I need SNS if I already use SQS? Only if more than one service needs each message. SNS then copies each message to one SQS queue per service, each with its own retries and dead-letter queue.
Does any message queue guarantee exactly-once delivery? Not end to end. SQS FIFO, Google Pub/Sub and Kafka transactions prevent duplicates within limits, but emails or payments still need an idempotent consumer.