c6Z8

Design a payment backend for an e-commerce application (e.g. Amazon.com). Handles all money movement.

Step 1 - Understand the Problem and Establish Design Scope

Requirements:

• Payment backend for e-commerce (Amazon-like)
• Supports credit cards, PayPal, bank cards (focus on credit cards for interview)
• Uses third-party payment processors (Stripe, Braintree, Square) — NOT direct processing
• Does NOT store credit card data internally (security/compliance)
• Global, but single currency assumed
• 1 million transactions/day
• Supports pay-in flow (money from buyers to platform) and pay-out flow (money to sellers)
• Reconciliation required (fix inconsistencies between internal + external services)

Non-functional: reliability/fault tolerance, reconciliation process (async verification across systems).

Back-of-the-envelope: 1M transactions / 86,400 sec ≈ 10 TPS. Low throughput → focus is on correctness, not raw scale.

Step 2 - Propose High-Level Design and Get Buy-In

Pay-in flow #

Figure 1: Simplified pay-in and pay-out flow

Figure 2: Pay-in flow

Components:

• Payment service: accepts payment events, coordinates process. First runs risk check (AML/CFT compliance, fraud detection — typically third-party).
• Payment executor: executes single payment order via PSP. One payment event may contain multiple payment orders.
• PSP (Payment Service Provider): moves money from buyer's credit card to platform's bank account.
• Card schemes: Visa, MasterCard, Discover — process credit card operations.
• Ledger: financial record (debit buyer $1,creditseller$1). Used for post-payment analysis (revenue, forecasting).
• Wallet: merchant account balance. Records total amount paid per user.

Pay-in flow steps:

1. User clicks "place order" → payment event → payment service
2. Payment service stores event in DB
3. Single checkout may split into multiple payment orders (products from multiple sellers). Payment service calls payment executor for each.
4. Payment executor stores payment order in DB
5. Payment executor calls external PSP to process credit card
6. On success: payment service updates wallet (seller balance)
7. Wallet server stores updated balance
8. Payment service calls ledger
9. Ledger appends new entry

Payment service APIs #

POST /v1/payments — Execute payment event. Params: buyer_info, checkout_id (globally unique), credit_card_info (encrypted or token), payment_orders list. Each payment_order: seller_account, amount (string, not double), currency (ISO 4217), payment_order_id (globally unique, used as PSP dedup/idempotency key).

Why string for amount: different protocols/hardware have different numeric precision → rounding errors. Numbers can be extremely large (Japan GDP ~5×10¹⁴ yen) or small (satoshi = 10⁻⁸ BTC). Parse to number only for display/calculation.

GET /v1/payments/{:id} — Get payment order execution status by payment_order_id.

Data model #

Storage choice factors: proven stability (5+ years in big financial firms), monitoring/investigation tools, DBA job market maturity. → Traditional relational DB with ACID over NoSQL/NewSQL.

Payment event table:

Payment order table:

Status states: NOT_STARTED → EXECUTING → SUCCESS / FAILED. On SUCCESS: update wallet (wallet_updated = TRUE) → update ledger (ledger_updated = TRUE). All orders under same checkout_id succeed → is_payment_done = TRUE. Scheduled job monitors in-flight orders, alerts on threshold exceeding.

Key insight: Pay-in moves money from buyer's credit card → platform's bank account (not seller's). Seller gets money via pay-out flow when conditions met (e.g. product delivered).

Double-entry ledger system #

Every transaction recorded in two ledger accounts with same amount. Sum of all entries = 0. One cent lost = someone gains a cent. End-to-end traceability. See Square's immutable double-entry accounting DB.

Hosted payment page #

Companies avoid storing credit cards (PCI DSS compliance burden). PSP provides hosted page (widget/iframe for web, SDK for mobile). PSP collects sensitive data directly — never reaches our payment system.

Figure 3: Hosted pay with PayPal page

Pay-out flow #

Similar to pay-in but uses third-party pay-out provider (e.g. Tipalti) to move money from platform bank account → seller bank account. Additional bookkeeping/regulatory requirements.

Step 3 - Design Deep Dive

PSP integration #

Two approaches:

1. Direct API: company stores payment info, develops payment pages. PSP connects to banks/card schemes.
2. Hosted payment page (common): PSP provides page, collects/stores card data. Company avoids PCI DSS.

Hosted payment flow:

Figure 4: Hosted payment flow

1. Client clicks "checkout" → payment service with payment order info
2. Payment service → PSP: payment registration request (amount, currency, expiration, redirect URL, nonce = payment_order_id for exactly-once registration)
3. PSP returns token (UUID, uniquely identifies registration)
4. Payment service stores token in DB
5. Client displays PSP-hosted payment page (Stripe JS library). Needs: token + redirect URL.

Figure 5: Hosted payment page by Stripe

6. User fills card details on PSP page → clicks pay → PSP processes
7. PSP returns payment status
8. Browser redirected to redirect URL (status appended as query param)
9. Async: PSP calls payment service via webhook with payment status. Payment service updates payment_order_status.

Reconciliation #

Async communication → no guaranteed delivery. Reconciliation periodically compares states across related services — last line of defense.

Figure 6: Reconciliation

Nightly: PSP/banks send settlement file (bank balance + daily transactions). Reconciliation system parses and compares with ledger.

Also used internally (ledger vs. wallet divergence detection).

Mismatch handling:

1. Classifiable + automatable: auto-fix via code
2. Classifiable, not automatable: manual fix by finance team (job queue)
3. Unclassifiable: special queue, finance team investigates

Handling payment processing delays #

Some payments take hours/days (PSP risk review, 3D Secure Authentication). PSP handles long-running requests:

• Returns pending status to client. Client shows pending + check-status page.
• PSP notifies via webhook on status change (or payment service polls PSP).

Three-party consistency (client, payment service, PSP) requires idempotency + reconciliation.

Communication among internal services #

Synchronous (HTTP):

• Cons: low performance (one slow service impacts all), poor failure isolation, tight coupling, hard to scale

Asynchronous (message queue/Kafka):

• Single receiver: shared message queue. Once processed, message removed. Figures 7-8: Message queue, single receiver

• Multiple receivers: Kafka. Same message consumed by multiple services (payment, analytics, billing). Figure 9: Multiple receivers

Sync = simpler but poor scalability. Async = trades simplicity for scalability + failure resilience. For large-scale payment systems with many dependencies → async preferred.

Handling failed payments #

Track payment state: definitive state at any stage, persisted in append-only table.

Retry queue + dead letter queue:

Figure 10: Handle failed payments

1. Check if failure is retryable
   • Retryable → retry queue
   • Non-retryable (invalid input) → error DB
2. Consume retry queue, retry transactions
3. If fails again:
   • Retry count < threshold → back to retry queue
   • Retry count ≥ threshold → dead letter queue (debug/inspect)

Exactly-once delivery #

Exactly-once = at-least-once (retry) + at-most-once (idempotency).

Retry (at-least-once) #

Figure 11: Retry

Retry strategies: Immediate, Fixed intervals, Incremental intervals, Exponential backoff (1s → 2s → 4s → ...), Cancel. Use exponential backoff for persistent network issues. Provide Retry-After header with error codes.

Idempotency (at-most-once) #

Client generates UUID as idempotency key → HTTP header: <idempotency-key: key_value>. Stripe/PayPal recommended.

Scenario 1 (double click): idempotency key = shopping cart ID pre-checkout. Second request seen before → return previous request's status. Concurrent duplicates → 429 Too Many Requests.

Figure 12: Idempotency

Implementation: DB unique key constraint. Insert → success = new request. Insert fails (PK exists) = duplicate → reject.

Scenario 2 (PSP processed but response lost): nonce (payment_order_id) → PSP → token. Token uniquely maps to payment order. Re-send same token → PSP recognizes duplicate, returns previous execution status.

Consistency #

Stateful services involved: payment service, ledger, wallet, PSP, DB replicas.

• Internal services: exactly-once processing
• Internal ↔ External (PSP): idempotency + reconciliation (never trust external system fully)
• Replica lag: Option 1: read+write from primary only (wastes replicas). Option 2: consensus algorithms (Paxos, Raft) or consensus-based DBs (YugabyteDB, CockroachDB).

Payment security #

Step 4 - Wrap Up

Designed payment system: pay-in/pay-out flows, PSP integration (hosted page), reconciliation (settlement files), async communication patterns, failure handling (retry + dead letter queues), exactly-once delivery (retry + idempotency), consistency, security. Additional considerations: monitoring, alerting, debugging tools, currency exchange, regional payment methods, cash payments (India/Brazil), Google/Apple Pay integration.