Design a Payment System

8 pm on the first night of a festive sale. Meena orders a ₹1,000 pressure cooker and approves the UPI request with her PIN. The app spins for 20 seconds, then says "Payment failed". She pays again, and the order goes through. Next morning her bank shows two debits of ₹1,000. The first payment had worked: the marketplace's server stopped waiting for the answer and guessed. Multiply Meena by thousands of buyers on a busy night and you get angry customers, a pile of refunds and books that no longer add up. A payment system that never guesses is what this chapter builds.

You are building the payment system for a large online marketplace in India. Buyers pay by card, UPI or the wallet balance they keep with you. The marketplace keeps a fee, pays sellers every day, and refunds buyers when needed. It must never charge twice, never lose a rupee, and always be able to show an auditor where every rupee went. One order runs through every page: Meena's ₹1,000 order ord_512, paid by UPI, from seller S-17.

Meena's double debit happened in the part of the marketplace that handles money, so start by saying exactly what that part must do. A marketplace sits between buyers and sellers like a shop counter between customers and suppliers: the buyer pays the marketplace, the marketplace keeps a fee, and the seller is paid later. The payment system is the part that talks to the world of banks and keeps the books.

Here are its requirements. The non-functional rows are the ones that shape the design.

Speed is missing from that list on purpose. The bank's side of a payment takes seconds, because the buyer types an OTP or a PIN, so nobody notices our 100 ms. Correctness beats latency in every trade-off in this chapter: when the system is unsure, it waits and checks rather than guessing.

Before designing anything, follow Meena's ₹1,000 out of her account. Our service never moves money itself. It asks a payment gateway to collect it: a company that connects merchants to card networks, banks and UPI, and reports each result back. Think of it as a courier that carries our request to her bank and brings the answer home.

For UPI, the gateway's request reaches Meena's UPI app, she approves it with her PIN, and the result flows back through the banks. Until then the payment is pending: she may still be finding her phone, or a bank may be slow. The real money reaches our bank later, usually in one lump sum a day, with a settlement file listing every payment and fee inside it.

Cards and our own wallet behave differently, in ways that matter to the design:

Only the wallet, which lives in our own ledger, is never unknown; every outside method can leave us waiting for an answer.

Card numbers stay with the gateway. Its card form, a hosted page or part of its app SDK, sends the number straight from the buyer's device to the gateway, which returns a token: a stand-in id, useless anywhere else, that we use to charge a saved card. That keeps most of our system out of scope for the card industry's security standard, PCI DSS. Indian rules also limit who may store card numbers, so check them before designing saved cards.

How busy is this system, and what does a mistake cost? Assume about 20 lakh paid orders a day at an average of ₹1,000, like Meena's, with big sales a few times a year.

Here is the full set of numbers:

Payments are a correctness problem, not a scale problem. A thousand payments a second fits on a handful of database machines, but every wrong rupee is an angry customer or an audit finding. The numbers settle four design questions:

• Transactions wherever money changes. At this rate, relational transactions are affordable, and they make "both or neither" easy.
• The bank sets the pace. Our work takes milliseconds; an OTP or a UPI PIN takes the buyer seconds or minutes. So paying is asynchronous: start, let the buyer approve, then learn the result.
• Hot accounts, not hot tables. Every payment credits our fee account, and a big seller may get 200 payments a second in a sale. Those single rows need care (the hot accounts page).
• Small data, kept for years. Older years move to cheaper storage; check the retention rules for your business.

With the scale known, define how Meena's payment starts. The order service creates payments, not the buyer's phone: it knows the true amount, and a phone can be tampered with. The phone only shows the gateway's card page or opens the UPI app. Every write carries an idempotency key: a unique name for one intended action, like a receipt number written down before money changes hands, so a repeat can be recognised.

POST /v1/payments                     (called by the order service)
     Idempotency-Key: ord_512:pay:1
     { orderId: "ord_512", amount: 100000, method: "upi" }
                      -> 201 { paymentId: "pay_81", status: "pending",
                               next: "approve in your UPI app" }
GET  /v1/payments/pay_81              -> 200 { status: "captured", ... }
POST /v1/payments/pay_81/capture      Idempotency-Key: pay_81:capture
                                      -> 200 { status: "captured" }
POST /v1/payments/pay_81/void         -> 200 { status: "voided" }
POST /v1/payments/pay_81/refunds      { amount: 40000 }
     Idempotency-Key: ord_512:refund:1 -> 201 { refundId: "rf_9", status: "pending" }
POST /v1/gateway-events               (sent by the gateway, signed)
     X-Signature: sha256=9c1f...      -> 200 once safely stored
GET  /v1/sellers/s_17/balance         -> 200 { payable: 9800000 }

Amounts are integers in paise: ₹1,000 is 100000. Floating-point numbers cannot hold most decimal fractions exactly, so money is never a float.

Four rules make the keys work:

• One idempotency key per intended action, made and stored before the first try, so even a caller that crashed retries with the same key. A new attempt after a decline gets a new key (ord_512:pay:2).
• The server stores each key with a hash of the request and the response. A repeat gets the saved response; the same key with a different amount is rejected; a repeat that arrives while the first is still running is told to retry shortly (409).
• A second guard in the data: (order_id, attempt) is unique, so even after a key expires, one attempt can never become two payments.
• pending is a real answer. The API never turns "not known yet" into "failed".

Where is Meena's payment recorded? Two kinds of data sit here, in separate databases. Payment records track each attempt to collect money and change state as results arrive, like a parcel tracker. Ledger entries record money itself, like lines in a bound account book: written once and never changed.

Each kind of data sits where its guarantees are cheapest:

A payment row changes; a ledger entry never does. A mistake is fixed with a new posting that reverses the old one, so the books show both the error and the fix, which is what auditors want. Refunds and payouts follow the same pattern in tables of their own. At our peak one primary per database copes, and payments can later be split by payment_id.

Meena's payment row needs one field everyone trusts: its status, the single source of truth about her money. Like a parcel tracker, it only moves forward along allowed arrows. The difficulty is that five sources can move it: the API, the gateway's direct reply, its webhook, the status checker and reconciliation, and their messages may arrive late, twice or out of order.

These are the states and the arrows between them:

Every arrow is one conditional update: change the row only if it is still in a state the arrow starts from, then check how many rows changed. It is the ticket booking chapter's seat trick, applied to money.

BEGIN;
UPDATE payments
   SET status = 'captured', version = version + 1
 WHERE payment_id = 'pay_81'
   AND status IN ('pending', 'authorised');
-- 1 row:  insert a payment_events row and an outbox row, then COMMIT
-- 0 rows: ROLLBACK and read the row
--         already captured: a repeat, so do nothing
--         failed or voided: a conflict, handled below

Every source calls this one function, so the order in which results arrive does not matter.

Three consequences follow:

• A late failure never beats a success. failed can only follow created or pending, so a failure message after a capture changes nothing except raising an alert.
• A late success after a failure is real money. If the gateway reported a UPI payment as failed or expired and a success arrives later (rare, but it happens), a separate, audited path records the capture, then attaches it to the order if still open, or refunds it automatically.
• History and events cannot drift: the status change, its payment_events row and its outbox row commit together.

With the states defined, here is the machinery that moves them. The design has a fast front half that starts payments and hears results, and a careful back half that records money. An outbox row, written in the same transaction as each status change and later published to a queue, joins the two, so a slow ledger never blocks a checkout and a crash never loses a result.

Meena's payment changed state only on the gateway's signed message, and the ledger heard about it through an outbox row committed in the same transaction. When the ledger stopped, events simply waited: nothing in the front half waits for the back half.

Each box has one job:

• Payment service: stateless servers owning the state machine, with one adapter per gateway, so adding a gateway is not a rewrite.
• Webhook receiver and status checker: two more doors into the same conditional update; the checker runs a delayed task per pending payment (the task scheduler chapter).
• Outbox relay and event queue: at-least-once delivery (the message queue chapter), so every consumer dedupes.
• Ledger: its own service; each posting has a unique reference such as pay_81:capture, so an event delivered twice posts once.
• Reconciliation and payouts: nightly batch jobs, covered later.

Meena's checkout touches more than payments. Three services each own their data: orders, stock and payments, and no single database transaction covers them. A two-phase commit, where every participant locks its rows until all agree, would hold the cooker's stock row while Meena types her PIN, and the bank cannot join one anyway.

Think of booking a holiday yourself: you book the train, then the hotel, and if the hotel is full you cancel the train. Checkout works the same way, as a saga: a series of local transactions, each with a compensation that undoes its effect if a later step fails. An orchestrator in the order service stores each order's current step and calls the next one (the publish-subscribe chapter compares this orchestration with choreography), so support can see where any order is stuck.

What the saga does when something goes wrong depends on the step:

In every row the order either moves forward or is undone cleanly; nothing is left half done.

Back to Meena's payment. The payment service asks the gateway to collect ₹1,000, and 10 seconds pass with no answer. As the abstractions chapter showed, the request may never have arrived, the reply may have been lost after the money moved, or a bank may just be slow. The service cannot tell which, so it must assume neither success nor failure.

Three habits made that safe: pay_81 was saved as pending before the call, its key went to the gateway so the retry could not debit twice, and every result used the same conditional update. A timeout means unknown, never failed, so Meena's screen should say "Confirming your payment. Please don't pay again."

The same key rule holds at every hop that can repeat a request:

Each hop dedupes on its own key, so a repeat anywhere replays a result instead of moving money. A gateway without keys gets no retried charges, only status checks.

A payment's status says whether money moved; the next question is where every rupee went. A balance column that payments add to and subtract from cannot explain itself: if it is wrong, nobody can say why. Picture an account book in which every rupee is written twice, once where it came from and once where it went, so the pages must always add up. A double-entry ledger works that way: every movement of money is a posting of at least two entries whose debits equal their credits, so a bug cannot create or lose money unseen.

Debit and credit are the ledger's two columns. A debit increases accounts for what we own or spend: our bank balance, money the gateway owes us, fees we pay. A credit increases accounts for what we owe or earn: money owed to sellers, our fee income. Here is Meena's ₹1,000 with our ₹20 fee, followed to seller S-17. Suppose the gateway's fee on this payment is ₹5; real fees depend on the method and the contract.

After all three, the gateway owes nothing, the seller is owed nothing, and the ₹15 left in the bank is our fee minus the gateway's. Each posting balanced on its own, so the whole book balances.

A balance is never stored as truth: it is the sum of an account's entries. We cache it on the account row for speed, and check every night that each cache equals its sum and that all debits equal all credits. Mistakes are fixed with new postings that reverse old ones, never by editing. One problem remains: what happens when a thousand postings a second touch the same account?

At a sale opening, every one of the 1,000 payments a second credits our fee income account, and a big seller may receive about 200. Entries are inserts, which never wait for each other. A cached balance, though, is one row that every posting must lock and update, and one row manages roughly a few hundred to a few thousand updates a second. Think of a sale with a single cash counter: the queue forms at the counter, not in the aisles.

Two fixes, often combined:

• Keep running balances only where money leaves. A wallet debit checks its balance in the same statement (... WHERE balance >= 50000), and a buyer pays a few times a day at most. Fee income needs no balance check, so it can be summed when needed.
• Split hot accounts, as the sharded counters chapter split a like count. Give fee income 16 sub-accounts and credit a random one per posting: about 63 updates a second each. Its balance is the sum of the 16; a big seller's account splits the same way, and a payout debits each part.

If one ledger database is not enough, shard it by seller and give every shard its own sub-accounts for the house accounts, so a capture still commits on one machine.

The cost lands on reads: a balance read sums 16 rows instead of one. A seller with a few orders a day needs none of this, so split only the accounts that are actually hot, and raise the count before a big sale rather than during it.

The ledger is now right by construction. What remains is proving that it matches what the gateway and the bank say happened.

A status check settled Meena's payment after its webhook was lost. That is one of three ways we learn each result, each a safety net for the one before: like a doorbell, a call to the courier, and the monthly statement.

They differ in speed and in how they fail:

Each later net is slower but harder to fool, so the slowest, the settlement file, has the final word.

Webhooks are untrusted until checked. The gateway signs each message with a secret only it and we know, commonly an HMAC of the body. We recompute and compare, reject old timestamps so a copy cannot be replayed, and check the amount. Only after storing the message and applying the conditional update do we reply 200, so any error makes the gateway resend.

Reconciliation is the nightly audit. Each gateway sends a file of everything it settled the day before, and our bank statement shows the lump sum that arrived. A batch job matches all three.

Most mismatches are timing: a payment captured at 11:59 pm may settle a day later. The dangerous ones are money we hold but never recorded, because a buyer paid for something we may not deliver. Every day closes with each rupee matched, or in the review queue with an owner.

Every fix keeps two promises: nobody pays twice, and every rupee stays accounted for. When a part fails, payments wait; they never guess.

Read down the Fix column and one habit repeats: when unsure, keep the payment pending and let a later safety net settle it.

The design made six choices, and each has a price. Here they are side by side.

The highlighted row is the heart of the chapter: a design that says "pending" can always become correct later, but one that guesses "failed" must find the buyer and give the money back.

What dominates the bill: not servers. Gateway fees, charged per payment and usually highest on cards, cost far more than the infrastructure; within our own costs, the synchronously replicated databases and years of kept records lead.

Where payment data may live. Indian rules require payment data to be stored in India. The details, such as which data counts, whom the rules bind and what may be processed abroad, have changed over time, so check the current rules with your compliance team. The design consequences are stable: both regions are Indian, so failover never leaves the country; backups, replicas, logs and analytics copies stay in India too; vendors that receive payment details must comply as well; and masking card numbers and UPI ids in logs leaves less data to protect.

Optional: the flows that send money back out. Suppose Meena's cooker arrives with a cracked lid, and support refunds ₹400 of her ₹1,000.

Refunds reuse everything above: each is its own record with its own key (ord_512:refund:1) and the same states, since its gateway call can time out too, and refunds appear in the settlement file as negative lines. One conditional update keeps refunds from ever adding up to more than was paid, even when two agents click at once:

Its posting runs the capture backwards: debit the seller's payable (and fee income, if the fee is returned) and credit gateway receivable, since the gateway deducts the refund from its next settlement. A refund to Meena's wallet instead is an internal posting: instant, with no gateway fee. Large refunds need a second person's approval, and both names go in the posting.

Payouts run once a day. For each seller, sum the payable entries past the return window, keep back a small reserve, and create one payout row with a unique key such as po_s17_1016, so rerunning the job cannot pay twice. The posting moves the money from the seller's payable to a payouts in transit account, and the bank's confirmation moves it on to the bank.

Payouts inherit the rules of payments:

• Bank transfers can time out too: check the payout's status by its reference; never resend it under a new one.
• A failed transfer is reversed, not edited: a new posting returns the money to the seller's payable.
• Refunds after a payout can push a seller below zero; the reserve covers it, and future sales repay it.

Here is how the 40 minutes of design in a 45-minute slot map onto this problem.

Choose the deep dive from the question. "What if the bank never answers?" or anything about double charges: the timeout and the state machine. How you know a seller's balance is right, or payouts: the ledger. How you would notice missing money: reconciliation. If asked about data location, one sentence covers it: payment data, its backups and its logs stay in India, so both regions are Indian.

Interviewers often change the problem halfway. Try each one before reading the answer.