Handbooks / System Design / Chapter 11
Distributed Transactions and Consensus
7 pages · ~5 min✓ Reviewed
Builds on Microservices and Service Communication. Next up: IDs, Time and Ordering.
Distributed Transactions and Two-Phase Commit
1. The Problem
- One logical operation spans several databases or services
- It must commit everywhere or nowhere (atomic commit)
2. Two Phases
- Preparecoordinator asks all
- Votelog, then YES or NO
- DecideCOMMIT or ABORT
- Ackparticipants confirm
- Phase 1: participant durably logs its vote; a YES is a promise it can still commit
- Phase 2: all YES gives COMMIT; any NO or timeout gives ABORT
- Coordinator logs the decision before sending it
3. Blocking and Alternatives
Sagas and the Transactional Outbox
1. What a Saga Is
- Saga = sequence of local transactions, no global lock
- Each step has a compensating action that undoes it semantically
- Examples: refund payment, cancel reservation
2. Orchestration vs Choreography
| Orchestration | Choreography | |
|---|---|---|
| Control | Central orchestrator tells each service the next step and tracks state |
Exactly-Once Myths and Idempotency
1. Delivery Guarantees
| Guarantee | Failure | How |
|---|---|---|
| At-most-once | may lose | send, never retry |
| At-least-once | may duplicate | retry until ack |
| Exactly-once delivery | impossible | Two Generals |
Leader Election and Raft Basics
1. Leader Election
- One node is the leader and coordinates writes
- Safety: at most one leader per term (epoch)
2. Raft Roles and Quorum
- Followerno heartbeat in time
- Candidateterm+1, requests votes
- Leaderwins majority
- Time is split into numbered terms
- Election timeout is randomized so ties are rare
Distributed Locks, Fencing Tokens, ZooKeeper and etcd
1. Leases and Why They Break
- Lease lock: expires after a TTL, so a crashed holder can't block forever
bash
SET lock:job42 clientA NX PX 30000NX = only if absent, PX = TTL in ms
- GC pause or slow network: lease expires, client still thinks it holds it
- Result: two holders write at once
2. Fencing Tokens
- A gets token 33then GC pause
- B gets token 34writes OK
- A writes with 33rejected
Distributed Coordination Cheat Sheet
1. Atomic Commit and Sagas
| 2PC | Saga | |
|---|---|---|
| Flow | prepare/vote, then commit/abort | local transactions + compensations |
| Failure | blocks if coordinator dies | run compensations |
| Consistency | atomic | eventual, no isolation |
Check yourself
In 2PC, every participant voted YES, then the coordinator crashed before sending any decision. Can a participant safely abort to free its locks?
- No. Its YES was a promise, and others may already have received COMMIT.
- It stays in-doubt, holding locks, until the coordinator recovers and replays its logged decision.
A consumer does send_email(msg) then mark_done(msg.id). It crashes between the two lines. What happens on redelivery, and what is the fix?
- The email goes out twice, because delivery is at-least-once.
- The dedup row can't cover an external call. Pass
msg.idto the email API as an idempotency key.
send_email(msg) mark_done(msg.id)