Publish-Subscribe

Sunday night, a cricket final, and a food delivery app is taking five times its usual orders. That week the marketing team had launched a small new service: a free dessert on every first order. To hear about orders, it is called by the order service, which already calls billing, the restaurant's tablet, receipts and analytics one after another. Mid-match, the dessert service slows to two seconds a call. Every checkout waits on it, and hungry customers watch a spinner, although billing and the kitchens are fine. The fix is to stop calling everyone: announce each order once and let every interested service listen.

That pattern is publish-subscribe (pub-sub). The order service publishes one event, "order placed", to a topic: a named channel for one kind of event. Every service that cares has a subscription to the topic and receives its own copy. The publisher does not know or care who is listening, so adding the dessert service no longer touches checkout.

Five words carry the rest of this chapter:

The key word is subscription: each one keeps its own progress, which is what lets one slow subscriber fall behind without holding up the rest. The next page shows exactly that.

What actually happens to one event inside the broker? Think of a newspaper: it is printed once, and every subscriber gets a copy at their own door, read whenever they like. A topic works the same way. For each new event, the broker records a pending copy for every subscription, or, in the log-based brokers of the message queue chapter, stores the event once and gives each subscription its own offset into the log. Either way, each subscription has its own backlog.

During the final, the coupon service ships a bad deploy and stops working for a few minutes while orders keep arriving.

The publisher sent each event once and never waited. While coupons were down, billing carried on, and the coupon backlog waited in its own subscription until the service came back. That isolation is the main reason to use pub-sub: one slow or broken subscriber cannot hold up the others.

A topic and the queues of the message queue chapter look alike: both are brokers holding messages between services. So when do you use which? The whole difference is who gets each message. In a queue, each message goes to one consumer. In a topic, each message goes to every subscription.

The first row decides everything else: one reader per message makes a queue, every reader makes a topic.

The command and event row gives a practical test. A command asks one service to do one thing, and you expect it done once: "charge order 512" goes to a queue. An event states a fact that already happened and lets anyone react: "order 512 placed" goes to a topic. The order service should not need to know that a dessert service exists; it just announces the order.

The two also combine. Billing runs several instances, and they must share billing's copy of each event the way workers share a queue. Pub-sub between services, a queue within each service: the next page shows how that works.

If billing runs 6 instances, they must not each charge the customer. So the instances of one service join a consumer group that shares a single subscription, like a household sharing one newspaper: one copy per house, read by whoever picks it up. Coupons have their own group, analytics another. The rule fits in one line: every group gets every event; within a group, each event goes to one member.

In partitioned brokers, a topic is split into partitions, and each partition is assigned to one member of each group. A group therefore cannot keep more members busy than the topic has partitions; extra members sit idle. Here is a topic with 4 partitions read by three groups:

When an instance joins, leaves or crashes, the broker moves its partitions to the others; this is a rebalance. That group can pause briefly, and an event in progress during the move may be delivered again to the new owner, one more reason subscribers must handle repeats. Each new assignment gets a higher generation number, so a member that was replaced without noticing cannot save progress for partitions it lost, a form of fencing the coordination chapter explains.

A new subscription must also decide where to start. In most classic brokers it sees only events published after it was created. In a log-based broker a new group chooses: begin at the oldest event still kept, or only at new ones.

Groups decide who reads an event. How does the event actually reach them? Like a doorbell against a letterbox, there are two ways. With push, the broker calls the subscriber as soon as an event arrives, usually through a webhook: an HTTPS endpoint the subscriber exposes for the broker to call. With pull, the subscriber asks the broker for a batch whenever it is ready.

Pull gives you back-pressure for free, which is why push tends to give way to pull as volume grows. Long polling narrows the latency gap: the broker holds a pull request open for up to, say, 20 seconds until something arrives.

POST /hooks/order-placed HTTP/1.1
Host: coupons.internal.example
Content-Type: application/json
X-Event-Id: evt_01HZX4T9
X-Signature: sha256=9c1f...e07a

{"type": "order.placed", "key": "ord_512", "data": {"user_id": 42, "total": 450}}

HTTP/1.1 204 No Content

A push delivery. Any 2xx reply counts as the ack; an error or a timeout means the broker retries later. The signature lets the subscriber reject events that did not come from the broker.

Push is simpler to consume, but the broker then needs rules for a subscriber that errors or slows down: exponential backoff, a cap on deliveries in flight, and a dead-letter destination after many failures.

How long does an event live after its subscribers have read it? In a classic pub-sub broker, it is removed from a subscription once that subscriber acks it. In a log-based broker, each event is stored once and stays for a retention period, such as 7 days, whether or not anyone has read it. A subscription's "own copy" is really its own position in one shared log, like a newspaper archive where each reader keeps a bookmark.

Keeping events unlocks replay. After fixing a bug in analytics, rewind its offset by a day and recompute. A brand-new fraud service can start a week back and build its own data from history, without asking another team for an export. Retention costs disk, so size it. Take the order-events topic during the final, carrying every status change of every order:

A week of replay for a busy topic costs a few TB of replicated disk, cheap next to rebuilding a service's data by hand.

For topics that describe the current state of something, such as a restaurant's opening hours, compaction is an alternative to time-based retention. The broker keeps at least the latest event for each key and discards older ones, so the topic stays small but a new reader can still rebuild the full current state.

Replays and rebalances mean a subscriber will see repeats. What else can arrive wrongly? The message queue chapter's rules hold: delivery is at least once, and order holds at best within one partition, so choose the key by what must stay in order. Pub-sub adds a trap of its own: there is no order across topics. "Payment captured" on the payments topic can reach a subscriber before "order placed" on the orders topic.

{
  "event_id": "evt_01HZX4T9",
  "type": "order.placed",
  "key": "ord_512",
  "version": 1,
  "occurred_at": "2026-10-18T20:41:03Z",
  "data": { "user_id": 42, "restaurant_id": 881, "total": 450 }
}

A unique id for deduplication, a key for ordering, and the order's version, which rises with every change to the order.

Each of the three problems has a cheap defence on the subscriber's side:

The version check solves two rows in one line: "apply only if the event's version is greater than the stored version" drops both duplicates and stale events. Parking has a cost of its own: the subscriber needs a small store for parked events and a deadline, after which it asks the owner directly, so an event whose partner was lost does not wait forever.

That event shape, with its id, key and version, now has readers you may never meet. What happens when you need to change it? Once ten services subscribe to order.placed, its shape is a public API. Renaming a field breaks every subscriber you forgot about, and with pub-sub you may not know who they all are. Four habits keep changes safe:

• Name events as past-tense facts. order.placed and payment.failed describe what happened; place.order reads like a command.
• Add, never remove or rename. New optional fields are safe; old subscribers ignore them.
• Version breaking changes. Publish order.placed.v2 alongside v1 until every subscriber has moved over.
• Validate when publishing. A schema registry, a shared store of event schemas, rejects a malformed event before it reaches anyone.

A second design choice is how much data an event carries:

Thin events turn every publish into a burst of reads on the owner, so most teams choose a middle ground: a fat event with only the fields subscribers need, leaving out personal details such as phone numbers unless a subscriber truly needs them.

With events designed well, where does pub-sub earn its place? The same order-placed events serve very different jobs:

The cache invalidation case links back to the distributed cache chapter. Instead of every writer remembering to delete keys in every region, the menu owner publishes one change event, and a small invalidator in each region subscribes and deletes the matching keys. Keep a TTL as well, because an event can be late or lost.

Pub-sub is the wrong tool in three cases:

• You need an answer back: a login check or a price quote is a request, not an event.
• Exactly one service must do the job: use a queue, or a topic with a single group.
• Two services only, for good: a direct call may be simpler.

The third card hides the hardest case. When placing an order needs billing, the restaurant and the order service each to commit their own step, what happens if a later step fails after an earlier one is done?

On final night, order 512 spans three services, each with its own database: the order service records it, billing charges ₹450, and the restaurant accepts or rejects it. No single database transaction covers all three. A saga is the usual answer: a chain of local transactions, each committed in one service, each publishing an event that starts the next. It works like booking a trip: if the hotel falls through after the flight is paid, you cancel the flight for a refund; you cannot un-book it.

When a later step fails, the saga runs compensating actions: new transactions that undo the earlier steps' effects.

The customer was charged and then refunded: two real transactions, both on the statement. That is the price of a saga: its in-between states are visible, so design them on purpose, and make every compensation idempotent and retried until it succeeds. Order the steps so the likeliest failure comes first: asking the restaurant before charging, or holding the money and capturing it only after acceptance where the payment method allows, would have avoided this refund, at the cost of a few seconds' wait at checkout.

In the scene, no service was in charge: each reacted to the event before it. Is that the best way to run a saga? That style is choreography. It keeps services independent, but no single place shows where order 512 is stuck, and the flow exists only as a set of subscriptions spread across teams.

The alternative is orchestration: one coordinator, the orchestrator, sends each step and each compensation in turn and records where every order stands, with a timeout on each step. The advice flips with size. Two or three steps owned by different teams suit choreography. Past three or four steps, or when support staff need to answer "where is my order?", an orchestrator is usually easier to follow and to debug. The payments chapter runs a saga end to end, with a ledger, provider callbacks and reconciliation.

Either way, the saga's steps travel as events or commands through the broker, so everything earlier in this chapter applies to them: repeats, reordering and lag. The next part looks at what goes wrong when those problems meet production traffic.

Pub-sub moves problems around rather than removing them. Which failures should you plan for from the start? Each one below has a cheap defence if it is designed in early:

Most rows come down to one habit: give each subscriber its own limits and its own alarms.

Two of these deserve numbers. If a subscriber can be down for a long weekend, 3 days, and you keep events for 2, it comes back to a hole in its data, so retention must exceed the longest outage you intend to recover from. And fan-out multiplies load: a burst of 5,000 events a second on a topic with 8 subscriptions is 40,000 deliveries a second.

Pub-sub belongs in a design wherever one action triggers several independent reactions; draw the topic, name the subscriptions, and back each line with a number or a trade-off.