Payment System
7 min readc6Z8
Design a payment backend for an e-commerce application (e.g. Amazon.com). Handles all money movement.
Step 1 - Understand the Problem and Establish Design Scope
Requirements:
- Payment backend for e-commerce (Amazon-like)
- Supports credit cards, PayPal, bank cards (focus on credit cards for interview)
- Uses third-party payment processors (Stripe, Braintree, Square) — NOT direct processing
- Does NOT store credit card data internally (security/compliance)
- Global, but single currency assumed
- 1 million transactions/day
- Supports pay-in flow (money from buyers to platform) and pay-out flow (money to sellers)
- Reconciliation required (fix inconsistencies between internal + external services)
Non-functional: reliability/fault tolerance, reconciliation process (async verification across systems).
Back-of-the-envelope: 1M transactions / 86,400 sec ≈ 10 TPS. Low throughput → focus is on correctness, not raw scale.
Step 2 - Propose High-Level Design and Get Buy-In
Pay-in flow #

Figure 1: Simplified pay-in and pay-out flow

Figure 2: Pay-in flow
Components:
- Payment service: accepts payment events, coordinates process. First runs risk check (AML/CFT compliance, fraud detection — typically third-party).
- Payment executor: executes single payment order via PSP. One payment event may contain multiple payment orders.
- PSP (Payment Service Provider): moves money from buyer's credit card to platform's bank account.
- Card schemes: Visa, MasterCard, Discover — process credit card operations.
- Ledger: financial record (debit buyer 1,creditseller1, credit seller 1). Used for post-payment analysis (revenue, forecasting).
- Wallet: merchant account balance. Records total amount paid per user.
Pay-in flow steps:
- User clicks "place order" → payment event → payment service
- Payment service stores event in DB
- Single checkout may split into multiple payment orders (products from multiple sellers). Payment service calls payment executor for each.
- Payment executor stores payment order in DB
- Payment executor calls external PSP to process credit card
- On success: payment service updates wallet (seller balance)
- Wallet server stores updated balance
- Payment service calls ledger
- Ledger appends new entry
Payment service APIs #
POST /v1/payments — Execute payment event. Params: buyer_info, checkout_id (globally unique), credit_card_info (encrypted or token), payment_orders list.
Each payment_order: seller_account, amount (string, not double), currency (ISO 4217), payment_order_id (globally unique, used as PSP dedup/idempotency key).
Why string for amount: different protocols/hardware have different numeric precision → rounding errors. Numbers can be extremely large (Japan GDP ~5×10¹⁴ yen) or small (satoshi = 10⁻⁸ BTC). Parse to number only for display/calculation.
GET /v1/payments/{:id} — Get payment order execution status by payment_order_id.
Data model #
Storage choice factors: proven stability (5+ years in big financial firms), monitoring/investigation tools, DBA job market maturity. → Traditional relational DB with ACID over NoSQL/NewSQL.
Payment event table:
| Name | Type |
|---|---|
| checkout_id | string PK |
| buyer_info | string |
| seller_info | string |
| credit_card_info | depends on provider |
| is_payment_done | boolean |
Payment order table:
| Name | Type |
|---|---|
| payment_order_id | String PK |
| buyer_account | string |
| amount | string |
| currency | string |
| checkout_id | string FK |
| payment_order_status | string (enum) |
| ledger_updated | boolean |
| wallet_updated | boolean |
Status states: NOT_STARTED → EXECUTING → SUCCESS / FAILED. On SUCCESS: update wallet (wallet_updated = TRUE) → update ledger (ledger_updated = TRUE). All orders under same checkout_id succeed → is_payment_done = TRUE. Scheduled job monitors in-flight orders, alerts on threshold exceeding.
Key insight: Pay-in moves money from buyer's credit card → platform's bank account (not seller's). Seller gets money via pay-out flow when conditions met (e.g. product delivered).
Double-entry ledger system #
Every transaction recorded in two ledger accounts with same amount. Sum of all entries = 0. One cent lost = someone gains a cent. End-to-end traceability. See Square's immutable double-entry accounting DB.
| Account | Debit | Credit |
|---|---|---|
| buyer | $1 | |
| seller | $1 |
Hosted payment page #
Companies avoid storing credit cards (PCI DSS compliance burden). PSP provides hosted page (widget/iframe for web, SDK for mobile). PSP collects sensitive data directly — never reaches our payment system.

Figure 3: Hosted pay with PayPal page
Pay-out flow #
Similar to pay-in but uses third-party pay-out provider (e.g. Tipalti) to move money from platform bank account → seller bank account. Additional bookkeeping/regulatory requirements.
Step 3 - Design Deep Dive
PSP integration #
Two approaches:
- Direct API: company stores payment info, develops payment pages. PSP connects to banks/card schemes.
- Hosted payment page (common): PSP provides page, collects/stores card data. Company avoids PCI DSS.
Hosted payment flow:

Figure 4: Hosted payment flow
- Client clicks "checkout" → payment service with payment order info
- Payment service → PSP: payment registration request (amount, currency, expiration, redirect URL, nonce = payment_order_id for exactly-once registration)
- PSP returns token (UUID, uniquely identifies registration)
- Payment service stores token in DB
- Client displays PSP-hosted payment page (Stripe JS library). Needs: token + redirect URL.

Figure 5: Hosted payment page by Stripe
- User fills card details on PSP page → clicks pay → PSP processes
- PSP returns payment status
- Browser redirected to redirect URL (status appended as query param)
- Async: PSP calls payment service via webhook with payment status. Payment service updates
payment_order_status.
Reconciliation #
Async communication → no guaranteed delivery. Reconciliation periodically compares states across related services — last line of defense.

Figure 6: Reconciliation
Nightly: PSP/banks send settlement file (bank balance + daily transactions). Reconciliation system parses and compares with ledger.
Also used internally (ledger vs. wallet divergence detection).
Mismatch handling:
- Classifiable + automatable: auto-fix via code
- Classifiable, not automatable: manual fix by finance team (job queue)
- Unclassifiable: special queue, finance team investigates
Handling payment processing delays #
Some payments take hours/days (PSP risk review, 3D Secure Authentication). PSP handles long-running requests:
- Returns pending status to client. Client shows pending + check-status page.
- PSP notifies via webhook on status change (or payment service polls PSP).
Three-party consistency (client, payment service, PSP) requires idempotency + reconciliation.
Communication among internal services #
Synchronous (HTTP):
- Cons: low performance (one slow service impacts all), poor failure isolation, tight coupling, hard to scale
Asynchronous (message queue/Kafka):
-
Single receiver: shared message queue. Once processed, message removed.
Figures 7-8: Message queue, single receiver -
Multiple receivers: Kafka. Same message consumed by multiple services (payment, analytics, billing).
Figure 9: Multiple receivers
Sync = simpler but poor scalability. Async = trades simplicity for scalability + failure resilience. For large-scale payment systems with many dependencies → async preferred.
Handling failed payments #
Track payment state: definitive state at any stage, persisted in append-only table.
Retry queue + dead letter queue:

Figure 10: Handle failed payments
- Check if failure is retryable
- Retryable → retry queue
- Non-retryable (invalid input) → error DB
- Consume retry queue, retry transactions
- If fails again:
- Retry count < threshold → back to retry queue
- Retry count ≥ threshold → dead letter queue (debug/inspect)
Exactly-once delivery #
Exactly-once = at-least-once (retry) + at-most-once (idempotency).
Retry (at-least-once) #

Figure 11: Retry
Retry strategies: Immediate, Fixed intervals, Incremental intervals, Exponential backoff (1s → 2s → 4s → ...), Cancel. Use exponential backoff for persistent network issues. Provide Retry-After header with error codes.
Idempotency (at-most-once) #
Client generates UUID as idempotency key → HTTP header: <idempotency-key: key_value>. Stripe/PayPal recommended.
Scenario 1 (double click): idempotency key = shopping cart ID pre-checkout. Second request seen before → return previous request's status. Concurrent duplicates → 429 Too Many Requests.

Figure 12: Idempotency
Implementation: DB unique key constraint. Insert → success = new request. Insert fails (PK exists) = duplicate → reject.
Scenario 2 (PSP processed but response lost): nonce (payment_order_id) → PSP → token. Token uniquely maps to payment order. Re-send same token → PSP recognizes duplicate, returns previous execution status.
Consistency #
Stateful services involved: payment service, ledger, wallet, PSP, DB replicas.
- Internal services: exactly-once processing
- Internal ↔ External (PSP): idempotency + reconciliation (never trust external system fully)
- Replica lag: Option 1: read+write from primary only (wastes replicas). Option 2: consensus algorithms (Paxos, Raft) or consensus-based DBs (YugabyteDB, CockroachDB).
Payment security #
| Problem | Solution |
|---|---|
| Eavesdropping | HTTPS |
| Data tampering | Encryption + integrity monitoring |
| MITM attack | SSL + certificate pinning |
| Password storage | Salted hashing |
| Data loss | Multi-region DB replication + snapshots |
| DDoS | Rate limiting + firewall |
| Card theft | Tokenization (store tokens, not real numbers) |
| PCI compliance | PCI DSS standard |
| Fraud | Address verification, CVV, user behavior analysis |
Step 4 - Wrap Up
Designed payment system: pay-in/pay-out flows, PSP integration (hosted page), reconciliation (settlement files), async communication patterns, failure handling (retry + dead letter queues), exactly-once delivery (retry + idempotency), consistency, security. Additional considerations: monitoring, alerting, debugging tools, currency exchange, regional payment methods, cash payments (India/Brazil), Google/Apple Pay integration.