Payment System for India

The payment backend for a marketplace with 3 lakh sellers. A buyer pays once for a cart from several sellers, mostly by UPI. Every rupee is charged once, recorded in a double-entry ledger, matched against the bank each morning, and paid out to sellers a week later.

Github | LinkedIn | WHOmI

ApnaBazaar is a fictional marketplace used as the running example: 20 lakh checkouts on a normal day, and 2,000 payment attempts a second when the festive sale opens at midnight.

Decisions at a glance

Requirements

Functional

• Take one payment for a cart by UPI (intent or QR), card, netbanking or wallet.
• Split each payment between the sellers in the cart, each with its own commission.
• Show the buyer success, failure or an honest “processing”.
• Refund cancellations and returns to the original method.
• Pay sellers into their bank accounts once the return window closes.
• Reconcile our books against the PA and the bank every day.

Non-functional

• Never charge a buyer twice for one checkout.
• Never lose a payment the bank has debited.
• Every rupee traceable, line by line, for auditors, tax filings and seller disputes.
• Checkout available through the festive-sale peak; side effects may lag but never drop.
• No card numbers stored, and all payment data kept in India.

Cash on delivery, foreign currency, and running our own payment switch are out of scope. Cash collected by couriers joins the same ledger later as its own journal.

Estimates

Even the sale-night peak is well within what a well-run relational database can take. The hard part is that the worst failures are silent: a debit with no order, an order with no debit, a seller paid twice. Nothing crashes, and the books are quietly wrong.

API

POST /v1/checkouts/chk_5TQ1/payments
Idempotency-Key: 6f1c2a0e-3b7d-4d8e-9a51-2c4f7e1b9d30     # made when Pay is tapped, reused on retries
{ "method": "upi_intent", "amount_paise": 179800 }       # a cross-check; the server recomputes the total
→ 201 { "attempt_id": "att_9KX2", "status": "PENDING",
        "next_action": { "type": "open_upi_app",
          "uri": "upi://pay?pa=apnabazaar@pabank&tr=att_9KX2&am=1798.00&cu=INR" } }


GET  /v1/payments/att_9KX2                          # the app polls while the buyer is in their UPI app

POST /v1/payments/att_9KX2/refunds                  # also takes an Idempotency-Key

POST /internal/webhooks/pa                          # PA results; HMAC signature checked first

All amounts are integer paise in a BIGINT: ₹1,798.00 is 179800. Floats can't hold money exactly, and a rounding error repeated over crores of payments stops the books from balancing. If the client's amount differs from the server's total, the API returns 409 and the app shows the new price.

Architecture

The system runs at three speeds. The buyer waits only on the payment path. Everything that follows a success happens asynchronously over Kafka. Real money moves between bank accounts on its own clock, a day or a week later.

Green boxes hold money or the record of it. Dashed boxes belong to banks and the PA, so we can’t fix them and can only check them against our own records.

Pay-in path

1. Meera taps Pay ₹1,798. The app creates an idempotency key for this tap and calls POST /v1/checkouts/chk_5TQ1/paymentsthrough the gateway.
2. The payment service recomputes the cart total, gets a quick verdict from the risk engine, and inserts attempt att_9KX2as CREATED.
3. The executor creates an order at the PA with att_9KX2as the reference. Meera's UPI app opens, she enters her PIN, and NPCI routes the debit from her bank. The attempt is now PENDING.
4. The PA sends a signed webhook: captured, with the bank’s UTR. The executor verifies the signature, then marks the attempt SUCCESSand writes an outbox row in one transaction.
5. Debezium publishes the outbox row to Kafka. The ledger posts the capture journal, the order service confirms both seller orders, and Meera gets an SMS.
6. The next day (T+1), the PA settles the day’s payments into our escrow account, minus its fees.
7. Seven days after delivery, once the return window closes, the payout service pays each seller their share from escrow.

BEGIN;
UPDATE payment_attempts
   SET status = 'SUCCESS', pa_payment_id = 'pay_Hx81', utr = '427815903311'
 WHERE attempt_id = 'att_9KX2' AND status IN ('CREATED', 'PENDING');
-- 0 rows: a duplicate or late webhook. Roll back, reply 200, do nothing else.
INSERT INTO payment_outbox (event_type, checkout_id, payload)
  VALUES ('PAYMENT_SUCCEEDED', 'chk_5TQ1', '{"attempt_id": "att_9KX2", …}');
COMMIT;

Why an outbox

If the executor updated Postgres and then published to Kafka as a separate step, a crash in between would lose the event: Meera is charged and her order is never confirmed. Writing the event in the same transaction as the status change means the two can’t disagree.

Why Kafka as a broadcast

One “payment succeeded” event has to reach the ledger, the order service and the notifier, so each is its own consumer group and sees every event. If the SMS provider is down, the SMS is late and the payment is fine. A chain of synchronous calls would fail the checkout instead.

Data model

-- PostgreSQL, Mumbai. All amounts are BIGINT paise.
checkouts        (checkout_id PK, buyer_id, amount_paise, status, expires_at)
seller_orders    (seller_order_id PK, checkout_id, seller_id, amount_paise, commission_paise)
payment_attempts (attempt_id PK, checkout_id, idempotency_key UNIQUE, method, amount_paise,
                  status, pa_payment_id UNIQUE, utr, response, updated_at)
payment_outbox   (id, event_type, checkout_id, payload)   rows deleted once published


-- Ledger database, append-only
ledger_journals  (journal_id PK, kind, created_at)        e.g. 'capture:att_9KX2'
ledger_entries   (entry_id PK, journal_id, account_id, direction, amount_paise)
account_balances (account_id PK, balance_paise, version)  a cache of the entries

Never charge twice

Meera taps Pay as her metro train enters a tunnel. The payment goes through, but the reply never reaches her phone. She sees an error and taps again. Without protection, the second tap is a brand-new purchase and she pays ₹3,596 for one cart.

Get Ajay Kotnala’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

Retries come from every direction, and we want them, because retrying is how payments don’t get lost. So we retry freely and make every retry harmless: exactly once is at-least-once delivery plus idempotent processing. Each hop that can repeat needs its own guard.

The last row is the case people miss. Meera’s UPI attempt hangs in pending, so she goes back and pays by card. Ten minutes later the UPI payment goes through too. Each attempt was idempotent on its own, but the checkout has been paid twice. The database stops it:

-- at most one successful attempt per checkout
CREATE UNIQUE INDEX one_success_per_checkout
  ON payment_attempts (checkout_id) WHERE status = 'SUCCESS';
-- the late UPI success violates it → record the money, refund it automatically

Spacing retries

PA calls and Kafka consumer retries use exponential backoff with jitter. Terminal errors, such as an invalid UPI ID or insufficient balance, are never retried. When we are overloaded we reply 429 or 503 with Retry-After, and our own app waits that long. On sale night, that header is what keeps a slowdown from becoming a retry storm.

Ledger

Idempotency makes each rupee move once. The ledger records every move, so that at any moment we can say where each rupee is and whose it is. Three rules:

• Every movement is a journal of two or more lines whose debits equal its credits.
• Entries are append-only. A mistake is fixed with a reversing journal, never an UPDATE.
• Balances are derived from entries. account_balances is a cache, rebuilt and compared every night.

The life of ₹1,798

ApnaBazaar takes 10% commission and, as the e-commerce operator, deducts TDS under Section 194-O at 0.1% of each sale. Here are the four journals:

Both seller payables and pa_receivable end at zero. What stays in escrow is ₹181.60: our commission plus the TDS owed to the government. A real fees journal also carries GST on the commission and the GST TCS that marketplaces collect, and card payments add a gateway-fee line (UPI carries zero MDR). The rule stays the same.

-- nightly proof: must return zero rows
SELECT journal_id FROM ledger_entries
 GROUP BY journal_id
HAVING SUM(CASE direction WHEN 'DEBIT' THEN amount_paise ELSE -amount_paise END) <> 0;

Hot accounts

Every capture debits pa_receivable. At 1,500 captures a second, every update waits on the same balance row's lock. Split it into pa_receivable:00 to :15, choose one by hash(attempt_id) % 16, and sum the sixteen when a report needs the total.

Pending UPI

Meera enters her PIN and her bank debits ₹1,798. Then the UPI app shows Transaction pending. A response timed out somewhere between her bank, NPCI and the PA’s bank. Most “money debited, order not placed” complaints start here, and the rule is to never guess.

Statuses only move forward along these arrows, and each arrow is a conditional update. A timeout never leads to FAILED; only the bank’s answer does.

• A timeout is not a failure. Marking it FAILED invites Meera to pay again, and then the first payment succeeds too.
• Webhooks first, status checks second. A background job checks pending attempts at growing intervals. NPCI limits how often one transaction’s status may be checked, so this is a few well-spaced checks, not a tight loop.
• Tell the buyer the truth. “Payment processing. Please don’t pay again. If money was debited, we’ll confirm your order or refund it automatically.” An SMS or WhatsApp message follows the final status.
• Hold stock for 15 minutes. A success that arrives later still confirms the order if stock is left, and is refunded automatically if not.
• Know the bank’s duty. RBI’s turnaround rules require a failed UPI debit to be reversed by T+1, with ₹100 a day in compensation after that. We store the UTR and failure reason so support can tell Meera exactly where her money is.

Reconciliation

Everything so far depends on messages arriving, and across crores of payments some won’t. Each morning a recon job matches three independent records of the same money on our reference, the UTR, the amount and the date.

Our ledger
What we believe happened, journal by journal.

PA settlement file
What the PA processed: each payment with our reference, its UTR, the fee and its settlement batch.

Escrow bank statement
What actually arrived: one credit per batch, equal to the batch minus fees.

Over time, the aim is to move cases up the table. Every unknown that gets explained becomes a rule, and every manual fix done often enough becomes automatic. Recon also runs inside our walls: every SUCCESS attempt needs a capture journal, and every capture journal needs a confirmed or refunded order.

Seller payouts

Pay-out is pay-in in reverse, and riskier, because money sent to the wrong account is hard to get back.

• When. An order’s money becomes payable when its return window closes, seven days after delivery. A daily run at 10 AM pays each seller their eligible balance in one transfer.
• Where. Each bank account is verified at onboarding with a penny drop: ₹1 over IMPS, and the account holder’s name the bank returns must match the seller’s KYC.
• Exactly once. The payout_id is the transfer reference, and the bank rejects duplicates. A payout that times out stays pending until its UTR shows up on the bank statement.
• Ledger first. Money moves from seller_payable to payout_in_transit before the bank call, and on to escrow_bank when the bank confirms. If the transfer bounces, a reversing journal puts it back.

CONTROL - A payout or ledger adjustment made by hand needs a maker and a different checker. Changing a seller’s bank account triggers a new penny drop and a 24-hour hold on payouts, because most payout fraud starts with that change.

Failures

Scaling

Payment status is always read from the primary. A lagging replica could show Meera PENDING a second after her payment succeeded, and she would pay again. Replicas serve dashboards only. Hyderabad holds an async copy for disaster recovery; a full regional failover may lose a few seconds of writes, and the next recon finds those payments in the PA's file.

Sale night

• Share the expected peak with both PAs weeks ahead, so they scale too.
• Scale stateless services and warm connection pools before midnight.
• Admit buyers through a waiting room at the gateway at a rate checkout can handle.
• 2,000 attempts a second at about 10 writes each is ~20,000 writes a second. If that’s too close to one primary’s limit, shard the payments database by hash(checkout_id) so each checkout stays on one shard.
• The ledger consumer posts 100 journals per transaction, turning ~12,000 row inserts a second into about 120 commits.

Security

• No card numbers. RBI’s tokenisation rules have barred merchants from storing card numbers since October 2022. The PA’s checkout collects card details, saved cards are network tokens, and the bank’s OTP step happens inside the PA’s flow. Our PCI DSS scope stays small.
• Never trust the redirect. The return URL after a card payment passes through the buyer’s browser and can be lost or forged. Only the signed server-to-server webhook, or a status check, confirms an order.
• Signed webhooks. The HMAC signature is checked before anything else is read, and old timestamps are rejected.
• Server-side amounts. The charge is always computed from the cart, never taken from the request.
• Data in India. RBI’s data-localisation rule keeps payment data in the country: Mumbai primary, Hyderabad recovery, backups and logs included. Tools hosted abroad see masked data only.
• Bots and fraud. Per-user and per-device rate limits at the gateway, a WAF, and a risk engine that checks velocity, device fingerprint and account age before charging.
• Insiders. Least privilege, maker-checker on money out, and an append-only ledger that keeps a full audit trail.

Naive vs. this design

ApnaBazaar, its buyers, sellers, IDs and amounts are examples. Regulatory figures (the TDS rate, IMPS and RTGS limits, RBI turnaround times) are summarised for design purposes; confirm current rules before relying on them.

ApnaBazaar, its buyers, sellers, IDs and amounts are examples. Regulatory figures (the TDS rate, IMPS and RTGS limits, RBI turnaround times) are summarised for design purposes; confirm current rules before relying on them.

System Design · Backend Development · Distributed Systems · SQLVsNoSQL· Database Design· Document Systems · Engineering· Payment System