A transition whitelist is the only way to change status
CAPTURED -> AUTHORIZED is rejected: state never goes backwards
System Design Case StudiesPage 5
PSP Integration and Retries
Calling the PSP from the Online Store: a call can end with an unknown outcome, so every retry must be safe to repeat.
1. Adapter and Timeouts
Payment servicetalks to one interface
PSP interfaceauthorize, capture, refund, status
Stripe adaptermaps to Stripe API
Adyen adaptermaps to Adyen API
One internal PSP interface; one adapter per provider (Stripe, Adyen)
Payment service never sees provider formats, so you can route or fail over
System Design Case StudiesPage 6
Idempotency Design
1. What Idempotent Means
Same request repeated → one effect and the same response
The client generates one UUID key per payment intent
Sent as an Idempotency-Key header on every attempt
Retries after a timeout reuse the key, so one charge results
2. Storage and Flow
sql
CREATETABLE idempotency_keys (
key TEXT PRIMARY KEY,
request_hash TEXT NOT NULL,
response JSONB,
status TEXT NOT NULL,
expires_at TIMESTAMPTZ NOT NULL
);
the PRIMARY KEY is the UNIQUE guard
System Design Case StudiesPage 7
Double-Entry Ledger
1. The Rule
Every transaction posts balanced entries: sum(debits) = sum(credits)
Money is moved between accounts, never created or destroyed
All entries of one payment event share one txn_id
Capture of 100.00
Debit
Credit
Customer Receivable / PSP Clearing
100.00
System Design Case StudiesPage 8
Webhooks and Reconciliation
1. Webhook Receiver
VerifyHMAC + timestamp
Persistraw event
Return 200within 2 s
Kafkaasync processing
Check the HMAC signature on the raw body with the shared secret
Reject a timestamp older than the 5 min replay window
Store the raw event before any parsing or business logic
Ack 200 in under 2 s; the ledger and order work happens after the ack
2. Duplicates, Order and Lost Events
System Design Case StudiesPage 9
Refunds and PCI Scope
1. Refund API and States
POST /v1/payments/{id}/refunds with its own idempotency key
Rule: sum(refunds) <= captured amount
Enforce it inside one DB transaction, not in app code
Partial refunds are allowed (several refunds per payment)
sql
BEGIN;
SELECT amount_minor FROM payments WHERE id = $1 FOR UPDATE;
SELECT COALESCE(SUM(amount_minor), 0) FROM refunds
WHERE payment_id = $1 AND status <> 'FAILED';
-- reject if refunded + new amount > capturedINSERT INTO refunds (payment_id, amount_minor, idempotency_key)
VALUES ($1, $2, $3);
COMMIT;
the row lock on payments serializes racing refunds
System Design Case StudiesPage 10
Scaling, Failure Handling, Trade-offs and Interview Questions
1. Scaling the Payments Store (Online Store)
Shard payments by order_id hash: shard = hash(order_id) % 8
Peak ~1,000 TPS / 8 shards = 125 avg, ~150 TPS per shard with skew headroom
One Postgres primary per shard, so every payment and its ledger rows stay in one ACID store
Read replicas serve status queries (GET /v1/payments/{id}), so writers keep the budget for writes
order_idhash % 8
Shard 1..8~150 TPS each
Primarywrites, ledger
Replicasstatus reads
System Design Case StudiesPage 11
Payment System Cheatsheet (Online Store)
1. Flow and Money
Createidempotency key
AuthorizePSP call
Capturetake the money
Ledger postbalanced entries
Webhook confirmPSP event
ReconcileT+1
Amounts are integer minor units (amount_minor BIGINT), never float
100.00 = 10000 minor units; no rounding drift
Debits = credits on every txn: 100.00 = 97.10 revenue + 2.90 fees
Corrections are reversing entries, never UPDATE or DELETE
2. Retries and States
System Design Case StudiesPage 12
Check yourself
The PSP call times out after 5 s. The retry code below runs on each attempt. Spot the bug.
The key is generated per attempt, so the PSP sees two different charges.
If the first call actually succeeded, the customer is charged twice.
Generate the key once per payment intent and send the same key on every retry.
A timeout means the outcome is UNKNOWN, so the retry must reuse the key (or query the PSP first).
for attempt in range(3):
key = str(uuid.uuid4())
resp = psp.authorize(amount, token, idempotency_key=key)
A payment is already CAPTURED when a delayed AUTHORIZED webhook arrives. What is the status afterwards?
The status stays CAPTURED, because state never moves backwards.
Functional: authorize, capture, void, refund
Query status of any payment
Receive PSP webhooks (async results)
Daily reconciliation against the PSP settlement file
Target
Value
Note
Create latency
p99 < 500 ms
excluding the PSP call
PSP call
timeout 5 s
timeout = unknown outcome
Webhook ack
< 2 s
return 200, process async
Double charge
none
exactly-once effect
Availability
99.99%
~52 min downtime/year
Lost money
zero
every cent traceable
Audit retention
7 years
payments and ledger
3. Scale and Storage Estimates (Online Store)
Orders to TPS
110M orders/dayone payment each
2÷ 86,400 sseconds per day
3~115 TPS averagesteady load
4x10 sale eventspeak multiplier
5~1,000 TPS peaksize for this
Item
Math
Result
Payment rows
~2 KB x 10M/day
~20 GB/day
Payment rows, yearly
20 GB x 365
~7 TB/year
Ledger entries
~4 per payment x 10M
~40M rows/day
Size for the peak
Capacity is planned for ~1,000 TPS, not the ~115 TPS average. The goal is an exactly-once effect (one charge), not exactly-once delivery.
Ledger service
Posts balanced, append-only entries
Webhook receiver
Verifies and queues PSP events
Reconciliation job
Daily match of PSP file vs payments vs ledger
Refund service
Reverses captured money, with its own idempotency key
2. Payment Flow
Happy path
1Ordercheckout creates it
2Create paymentidempotency key
3PSP authorizeadapter call
4Webhook confirmsevent queued
5Ledger postsbalanced entries
6Order paidstatus updated
The user-facing call returns PENDING quickly: p99 < 500 ms excluding the PSP
The final result arrives by webhook, or by polling the PSP if the webhook is lost
Sync PSP call: simple but blocks the user. Async: resilient, more states
3. Card Data and Data Stores
Browserhosted fields / redirect
PSPreceives the card
Tokenreturned to browser
Our serverstoken only
Card data goes browser -> PSP directly, never through our servers
The store only receives a token
Store
Holds
Why
Postgres
Payments, ledger
ACID transactions
Redis
Idempotency cache
Fast duplicate check
Kafka
Events, webhook queue
Durable, replayable, async
Remember
Create returns PENDING; the webhook (or polling) decides the outcome. The card never touches our servers, only a token.
API podsstateless
Postgres shardshash of order_id
Outbox tablesame DB transaction
Kafkaevents
Pods hold no state, so scale them horizontally
Shard key: merchant/order_id hash
Outbox row is written in the same transaction as the payment row
A relay publishes outbox rows to Kafka, so no event is lost
2. Payment Flow
POST /v1/payments
1Insert rowstatus INITIATED
2Call PSPsame idempotency key
3Store psp_refAUTHORIZED
4CaptureCAPTURED
3. Data Model and API
sql
CREATETABLE payments (
id UUID PRIMARY KEY, order_id TEXT NOT NULL,
amount_minor BIGINT NOT NULL, currency CHAR(3),
status TEXT NOT NULL, psp_ref TEXT,
idempotency_key TEXT NOT NULL UNIQUE,
version INT NOT NULL DEFAULT 0,
created_at TIMESTAMPTZ NOT NULL DEFAULT now());
money is integer minor units: 100.00 = 10000
Endpoint
Purpose
POST /v1/payments
create and authorize
POST /v1/payments/{id}/capture
capture an authorized payment
POST /v1/payments/{id}/refunds
refund, full or partial
GET /v1/payments/{id}
query current status
4. Failure and Metrics
Crash after the PSP call, before the DB write: row is stuck in INITIATED
A recovery job queries the PSP by idempotency key
It resolves the state: found means store psp_ref and advance, none means FAILED
Never assume failure, never charge again
Metric
Target / alert
Payment success rate
>= 97%
Create latency p99
< 500 ms excluding PSP
Stuck in PENDING
alert if > 15 min
PSP error rate
by error code
Remember
Never use floats for money: store amount_minor BIGINT. The UNIQUE idempotency_key plus the outbox keep the charge and the event exactly-once in effect.
From
Legal next states
INITIATED
PENDING, FAILED
PENDING
AUTHORIZED, FAILED, EXPIRED
AUTHORIZED
CAPTURED, VOIDED, EXPIRED
CAPTURED
PARTIALLY_REFUNDED, REFUNDED
PARTIALLY_REFUNDED
PARTIALLY_REFUNDED, REFUNDED
2. Guarding a Transition
Optimistic locking: check version and the expected status in the same UPDATE
0 rows updated = lost the race or illegal move, so reject; no blocking lock held
sql
UPDATE payments
SET status = 'CAPTURED',
version = version + 1WHERE id = ?
AND version = ?
AND status = 'AUTHORIZED';
racing writers: only one gets rowcount = 1
Append-only payment_events in the same DB transaction: full audit trail and replay
Never UPDATE or DELETE events; cause says why (webhook, API, job)
from_status / to_status avoid Postgres reserved words
3. Failures and Metrics
CAPTURED webhook before AUTHORIZED: fast-forward and write both events
Late AUTHORIZED after CAPTURED: no-op, log the event, never downgrade
Replaying the same webhook changes nothing: handlers are idempotent
Remember
State only moves forward. Lock with version + status, and log every move in payment_events.
Metric
Signals
Transition counts
traffic mix per state pair
Illegal-transition rejections
bugs or racing writers
Time spent per state
stuck PENDING payments
Auth expiry rate
uncaptured holds (~7 days)
Timeouts: connect 1 s, read 5 s
A timeout means UNKNOWN outcome: the charge may have succeeded
Never mark a timed-out call FAILED; resolve it by querying the PSP
2. Retries and Idempotency
Error
Retry?
Why
Network error / timeout
Yes
Outcome unknown, same key makes it safe
5xx
Yes
PSP-side fault, likely transient
429
Yes
Rate limited, back off first
4xx decline (card, funds)
No
Same request gets the same answer
Wait = 2^attempt s + small jitter, so 1 s then 2 s
Max 3 attempts, so only two waits
Jitter spreads retries so a recovering PSP is not hit in a burst
Send the same idempotency key on every retry: PSP dedupes, one charge results
http
POST /v1/payment_intents
Idempotency-Key: 7f3c9a52-1b4e-4d0a-9c1e-5a2b8e6d1f04
key is fixed per payment intent, reused on every attempt
Retry rule
Retry only network, 5xx, 429, always with the same key. Never retry a 4xx decline.
3. Circuit Breaker and Metrics
Breaker states
>50% errors over 30 safter 30 sprobe succeedsprobe fails
status is IN_PROGRESS or COMPLETED
expires_at gives a TTL of 24-48 h, then a cleanup job deletes the row
request_hash = hash of the request body, used to spot a changed request
First request with a new key
1Insert keystatus IN_PROGRESS
2Processcreate payment, call PSP
3Store responsestatus COMPLETED
4Duplicate arrivesreplay stored response
Duplicate seen
Code
Result
Same key, same body, COMPLETED
200
replay the stored response
Same key, same body, IN_PROGRESS
409
concurrent duplicate, retry later
Same key, different body
422
request_hash mismatch
3. Example: The Insert Decides
sql
INSERT INTO idempotency_keys
(key, request_hash, status, expires_at)
VALUES ($1, $2, 'IN_PROGRESS',
now() + interval '24 hours')
ON CONFLICT (key) DO NOTHING;
-- 1 row: we own the key, process it-- 0 rows: duplicate, read the row
one atomic statement, no check-then-write gap
Redis equivalent: SET key val NX EX 86400 is atomic too
On 0 rows, compare request_hash, then replay, 409 or 422
4. Common Mistakes
Check Redis, then write separately: two requests both pass the check, a race
Fix: atomic DB unique insert or SET NX
New key per HTTP attempt instead of per intent: dedupe never matches
That mistake turns every retry into a double charge
Remember
One key per payment intent, created once by the client and reused on every retry. The unique insert, not a prior read, decides who processes.
Merchant Revenue
97.10
Fees Payable
2.90
Total
100.00
100.00
2. Data Model and Corrections
sql
CREATETABLE ledger_entries (
id BIGSERIAL PRIMARY KEY,
txn_id UUID NOT NULL,
account_id BIGINT NOT NULL,
direction CHAR(1) NOT NULL,
amount_minor BIGINT NOT NULL,
currency CHAR(3) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT now());
direction is D or C; 100.00 is stored as 10000 minor units
Table is append-only: rows are only ever inserted
Balances are derived (SUM over entries) or cached per account
A mistake is fixed by a reversing entry, never UPDATE or DELETE
Gives immutability and a full audit trail
3. Posting Safely
BEGINone DB txn per txn_id
Insert legsall entries
Check balancedebits = credits
COMMITor roll back all
sql
ALTER TABLE ledger_entries
ADD CONSTRAINT one_leg_per_txn
UNIQUE (txn_id, account_id, direction);
a retried post hits the constraint, rolls back, nothing double-posts
Never edit history
A posted entry is never changed or deleted. Post a reversing entry instead, so the books always show what happened.
4. Metrics
Metric
Target
Ledger imbalance
0, alert on any nonzero
Posting latency p99
< 50 ms
Entries per payment
~4
The PSP retries with backoff for up to ~3 days, so duplicates are normal
Dedupe by event_id; the handler must be idempotent
Events arrive out of order: apply by state machine, never downgrade state
Lost webhook: poll the PSP status API for payments PENDING > 15 min
sql
INSERT INTO webhook_events (event_id, raw_body, received_at)
VALUES ($1, $2, now())
ON CONFLICT (event_id) DO NOTHING;
0 rows inserted means a duplicate: ack 200 and skip
3. Daily Reconciliation
3-way match by psp_ref and amount
1PSP settlement filesource of truth for money moved
2payments tableour status and amount_minor
3ledger entriesbalanced postings per txn
Mismatch
Meaning
Typical auto-fix
Missing in ours
PSP has it, we do not
Create from PSP data, then post ledger
Missing at PSP
We have it, PSP does not
Query PSP by idempotency key, then void or fail
Amount differs
Same psp_ref, different amount
Ticket for review, no auto-fix
Status differs
Our state is behind or wrong
Re-apply via state machine
4. Metrics
Metric
Target
Unmatched rate
< 0.01%
Webhook lag p99
< 5 s
DLQ depth
alert when above 0 and growing
Reconciliation completion
by T+1
Webhooks are not guaranteed
Webhooks can be lost, so polling catches it within minutes and daily reconciliation is the final safety net for zero lost money.
REQUESTEDrow inserted
PENDINGsent to PSP
SUCCEEDEDPSP confirms
FAILEDPSP rejects
Money reaches the customer after 5-10 business days (settlement)
SUCCEEDED means the PSP accepted it, not that the card is already credited
2. Refund Ledger Entries
Entry
Account
Direction
Original capture
PSP Clearing
debit
Original capture
Merchant Revenue
credit
Refund
Merchant Revenue
debit
Refund
PSP Clearing
credit
A refund reverses the original entries; it is a new append-only txn
PSP fees are usually not returned, so Fees Payable stays as posted
3. PCI DSS Scope
Never store the PAN, the CVV or full track data
Store only the PSP token + last4 + card brand
Hosted fields/iframe or redirect: card data goes browser -> PSP, never through your servers
SAQ A
SAQ D
Card data touches your servers
No
Yes
Integration
hosted fields / iframe / redirect
raw card handling
Controls
fewest (smallest scope)
~300+
4. Mistakes and Protection
Logging card data
Logging request bodies that contain card data or tokens pulls your logs into PCI scope. Log ids and last4 only.
2. Failure Handling
Outbox: no event lost, no event without a write
1One DB txnpayment row + outbox row
2Relayreads outbox
3Kafkaat-least-once
4Consumeridempotent
5DLQpoison event after retries
Outbox: the DB write and the event commit together, so a crash never leaves one without the other
Saga with compensation: order fails after authorize, so void the auth
DLQ: a poison event is parked after max retries, alerted on depth, replayed after a fix
Multi-AZ for DB, Kafka and pods; multi-PSP failover through the circuit breaker
3. Trade-offs
Sync PSP call
Async with webhook
Complexity
Simple, few states
More states: PENDING, UNKNOWN
User
Blocked up to the 5 s timeout
Returns PENDING fast
Resilience
PSP slowness hits checkout
Resilient, result by webhook or poll
Ledger
Analytics
Consistency
Strong, ACID, balanced
Eventual, from Kafka
Staleness
None allowed
Seconds to minutes is fine
Wrong value
Lost money
A stale dashboard
4. Interview Questions
Double charge on a timeout? A timeout is UNKNOWN: retry with the same idempotency key, and query the PSP first
Lost webhook? Poll the PSP for payments PENDING > 15 min, plus daily reconciliation as the safety net
Why a ledger, not just a payments table? Immutable audit trail, balanced entries, proof of where every cent went
Remember
Shard by order_id, publish through the outbox, void on failure, and never assume a timeout failed: ask the PSP.
Failure
Retry?
Why
Network error / timeout
Yes
Outcome is UNKNOWN, same key
5xx
Yes
PSP-side fault
429
Yes
Rate limited, back off
4xx decline
Never
Card refused; retry cannot help
Exponential backoff + jitter: max 3 attempts, waits of 1 s then 2 s + jitter
Send the same idempotency key on every retry: PSP dedupes, one charge
States only move forward; CAPTURED -> AUTHORIZED is rejected
Optimistic locking on version + status, plus an append-only event log
sql
UPDATE payments SET status='CAPTURED', version=version+1WHERE id=? AND version=? AND status='AUTHORIZED';
INSERT INTO payment_events
(payment_id, from_status, to_status, cause, at)
VALUES (?, 'AUTHORIZED', 'CAPTURED', 'psp_capture', now());
0 rows updated = a racing writer won; do not write the event
3. PCI and Targets
Tokens only: store PSP token + last4 + brand
Hosted fields or redirect give SAQ A (smallest scope)
Never log PAN or CVV, including request bodies
Target
Value
Peak load
~1,000 TPS
Availability
99.99%
Payment create latency
p99 < 500 ms
Unmatched in reconciliation
< 0.01%
Remember
Same key on retry, never retry a decline, never move state backwards, and ledger debits must equal credits.
The late event is applied idempotently as a no-op.
It is logged in payment_events for the audit trail.
It is not counted as an illegal-transition rejection, because that metric signals bugs or racing writers.
A capture of 100.00 posts: debit PSP Clearing 100.00, credit Merchant Revenue 100.00, credit Fees Payable 2.90. What is wrong, and how is it caught?
Debits total 100.00 but credits total 102.90, so the entries are unbalanced.
Revenue should be 97.10, so that credits 97.10 + 2.90 = 100.00.
Posting runs as one DB transaction that rejects any txn where sum(debits) != sum(credits).
The imbalance alert fires on any nonzero value, since the target is always 0.
A 100.00 payment is CAPTURED. A 60.00 refund has succeeded, and now a 50.00 refund request arrives. What happens?
It is rejected: 60 + 50 = 110, which is more than the 100.00 captured.
The check sum(refunds) <= captured runs inside a DB transaction, so two racing requests cannot both pass.
A refund of 40.00 or less would still be allowed, since partial refunds are fine.
Each refund carries its own idempotency key, so a retried 60.00 request is not refunded twice.
Key K was used for a 100.00 payment. The same key K now arrives with an amount of 120.00. What is returned?
422, because the stored request_hash differs from the new request's hash.
The stored response is not replayed, since this is a different request, not a duplicate.
A concurrent duplicate with the identical body would get 409 while the first is still IN_PROGRESS.
An identical body after completion gets the stored response replayed.