Handbooks / Microservices / Chapter 3
Data and Consistency
7 pages · ~5 min✓ Reviewed
Builds on Service Communication. Next up: Resilience Patterns.
Distributed Transaction Problems
1. Why ACID stops at the service
- One business operation spans several services
- Each service owns its own database
- No single ACID transaction can cover all of them
2. Two-phase commit (2PC)
- 1Preparecoordinator asks all
- 2Voteseach says yes or no
- 3Commit or abortall or nothing
- 2PC is blocking
- Coordinator crash: participants hold locks until it returns
3. Dual writes and CAP
- Dual write: DB update and broker publish are two steps
- Crash between them: DB and broker are out of sync
- Network partition: choose consistency or availability
- Most microservices choose availability
4. Retries need idempotency
bash
curl -X POST /charges \
-H "Idempotency-Key: order-42-charge"server stores the key and replays the first result
Retry without a key
A retry repeats the side effect, such as a double charge. Give every step an idempotency key.
Saga Pattern: Choreography vs Orchestration
1. What a saga is
- Saga = sequence of local transactions, one per service
- Each step has a compensating transaction
- If a later step fails, compensations semantically undo the earlier ones
2. Choreography vs orchestration
| Choreography | Orchestration | |
|---|---|---|
| Control | services react to events | central orchestrator sends commands |
| State | spread across services | tracked by orchestrator |
| Strength | loosely coupled | easy to follow |
| Weakness | hard to trace, cyclic dependencies | coupling point, keep it thin |
3. Example: payment fails
- Reserve stock
- Charge paymentfails
- Release stockcompensate 1
- Cancel ordercompensate 2
Outbox Pattern
1. The Outbox
- Write the business row and an outbox event row in the same local DB transaction
- Both commit or both roll back, so the dual-write problem is gone
- Publishing to the broker happens later, from the outbox table
- One DB transactionorder row + outbox row
- Relaypoll or CDC
- Brokerpublish event
- Mark sentafter publish
2. Relay and Delivery
| Polling | CDC (log tailing) | |
|---|---|---|
| How | Query unsent rows on a timer | Read the DB log, e.g. Debezium |
| Latency | Higher | Lower |
| DB load | Extra query load | None |
- Relay can crash after publish, before marking sent
- So delivery is at-least-once: duplicates happen
- Consumers must be idempotent
CQRS and Event Sourcing
1. CQRS
- Write model: commands change state
- Read model: queries over denormalized views
- Each is scaled and stored independently
- Projections consume write-side events, so reads lag writes: eventually consistent
- Command
- Write modelemits event
- Projectionlag here
- Read modelquery view
2. Event Sourcing
- Append-only log of immutable events is the source of truth
- State = fold (replay) of one aggregate's events
- Snapshot every N events: no full replay
- 1Load snapshotstate at event N
- 2Replay newer eventsN+1 onward
- 3Current state
3. Together and Gotchas
| CQRS | Event sourcing | |
|---|---|---|
| Without the other? | Yes | Rarely |
Eventual Consistency in Practice
1. Eventual Consistency
- No new updates → all replicas and read models converge to the same value
- Reads in between may be stale
- Gap = replication or projection lag
2. Living With It
- Read-your-writes: send that user's reads to the primary
- Or wait until the projection reaches the write's version
- At-least-once delivery → duplicates are normal
- Idempotent consumer: check a dedupe key first
sql
BEGIN; INSERT INTO inbox(msg_id) VALUES ($1) ON CONFLICT DO NOTHING; -- 0 rows inserted: duplicate, skip UPDATE ...; COMMIT;
dedupe insert and the update share one transaction
3. Choose the Pattern
| Scope | Use |
|---|---|
| One service's DB | Strong consistency (local ACID) |
| Across services | Saga |
| Few trusted XA resources | 2PC |
Not for simple CRUD
Cheat Sheet: Distributed Consistency Patterns
1. Problem to pattern
| Problem | Pattern |
|---|---|
| Cross-service atomicity | Saga |
| DB plus broker write | Outbox |
| Read/write scaling | CQRS |
| Full audit and replay | Event sourcing |
2. Saga choice and delivery
- Choreography: 2-4 simple steps
- Orchestration: complex or branching flows
- At-least-once delivery + idempotent consumers = effectively-once
3. Compensation rule
- Undo in reverse order
- Make every undo idempotent
- Never rely on rollback
- Payment fails
- Release stock
- Cancel order
4. Checklist
Check yourself
A service commits an order row, then crashes before publishing OrderCreated to the broker. What state is the system in, and what fixes it?
- Order exists, but no one hears about it: the DB and broker are out of sync.
- Fix: write an outbox row in the same transaction; a relay publishes it later.
The outbox relay publishes a row, then crashes before marking it sent. What happens on restart?
- The row is published again: delivery is at-least-once.
- Consumers must dedupe by message ID, ideally in the same transaction as their update.
Saga: reserve stock, charge payment, ship. Shipping fails. What runs, in what order?
- Refund payment, then release stock, then cancel order: reverse order.
- Each compensation must be idempotent, since it may be retried.