Payment System

Design a payment backend for an e-commerce application (e.g. Amazon.com). Handles all money movement.

Step 1 of 4: Understand the problem and establish design scope #

Goal: lock in requirements, constraints, and scale before designing.

Requirements #

Must have:

1. Payment backend for e-commerce (Amazon-like)
2. Credit cards, PayPal, bank cards: focus on credit cards for the interview
3. Third-party payment processors (Stripe, Braintree, Square): NOT direct processing
4. No internal credit-card storage (security/compliance)
5. Pay-in flow (buyers → platform) and pay-out flow (platform → sellers)

Nice to have / assumptions:

1. Global, but single currency assumed
2. 1 million transactions/day
3. Non-functional: reliability, fault tolerance
4. Reconciliation: async verification across internal + external services

Back-of-the-envelope: 1M transactions / 86,400 sec ≈ 10 TPS: low throughput → focus is on correctness, not raw scale.

Step 2 of 4: Propose high-level design and get buy-in #

Goal: agree on the flows and components before diving deep.

Pay-in flow #

Figure 1: Simplified pay-in and pay-out flow

Figure 2: Pay-in flow

Components:

Pay-in flow steps:

1. User clicks "place order" → payment event → payment service
2. Payment service stores event in DB
3. Single checkout may split into multiple payment orders (products from multiple sellers) → payment service calls payment executor for each
4. Payment executor stores payment order in DB
5. Payment executor calls external PSP to process credit card
6. On success: payment service updates wallet (seller balance)
7. Wallet server stores updated balance
8. Payment service calls ledger
9. Ledger appends new entry

Payment service APIs #

POST /v1/payments: execute a payment event. Params:

• buyer_info
• checkout_id (globally unique)
• credit_card_info (encrypted or token)
• payment_orders list: each: seller_account, amount (string, not double), currency (ISO 4217), payment_order_id (globally unique, used as PSP dedup/idempotency key)

GET /v1/payments/{:id}: get payment order execution status by payment_order_id.

Data model #

Storage choice factors: proven stability (5+ years in big financial firms), monitoring/investigation tools, DBA job market maturity → traditional relational DB with ACID over NoSQL/NewSQL.

Payment event table:

Payment order table:

Status flow: NOT_STARTED → EXECUTING → SUCCESS / FAILED. On SUCCESS: update wallet (wallet_updated = TRUE) → update ledger (ledger_updated = TRUE). All orders under the same checkout_id succeed → is_payment_done = TRUE. Scheduled job monitors in-flight orders, alerts when threshold exceeded.

Double-entry ledger #

Every transaction is recorded in two ledger accounts with the same amount → sum of all entries = 0. One cent lost = someone gains a cent. End-to-end traceability (see Square's immutable double-entry accounting DB).

Hosted payment page #

Companies avoid storing credit cards (PCI DSS compliance burden). The PSP provides a hosted page (widget/iframe for web, SDK for mobile) and collects sensitive data directly: it never reaches our payment system.

Figure 3: Hosted pay with PayPal page

Pay-out flow #

Similar to pay-in, but uses a third-party pay-out provider (e.g. Tipalti) to move money from platform bank account → seller bank account. Additional bookkeeping/regulatory requirements.

Step 3 of 4: Design deep dive #

Goal: harden the design: PSP integration, failure handling, consistency, security.

PSP integration: two approaches #

1. Direct API: company stores payment info, develops payment pages; PSP connects to banks/card schemes
2. Hosted payment page (common): PSP provides the page, collects/stores card data → company avoids PCI DSS

Hosted payment flow:

Figure 4: Hosted payment flow

1. Client clicks "checkout" → payment service with payment order info
2. Payment service → PSP: payment registration request (amount, currency, expiration, redirect URL, nonce = payment_order_id for exactly-once registration)
3. PSP returns token (UUID, uniquely identifies registration)
4. Payment service stores token in DB
5. Client displays PSP-hosted payment page (Stripe JS library): needs token + redirect URL

Figure 5: Hosted payment page by Stripe

6. User fills card details on PSP page → clicks pay → PSP processes
7. PSP returns payment status
8. Browser redirected to redirect URL (status appended as query param)
9. Async: PSP calls payment service via webhook with payment status → payment service updates payment_order_status

Reconciliation: the last line of defense #

Async communication → no guaranteed delivery. Reconciliation periodically compares states across related services.

Figure 6: Reconciliation

Nightly: PSP/banks send a settlement file (bank balance + daily transactions). Reconciliation system parses it and compares with the ledger. Also used internally (ledger vs. wallet divergence detection).

Mismatch handling:

1. Classifiable + automatable → auto-fix via code
2. Classifiable, not automatable → manual fix by finance team (job queue)
3. Unclassifiable → special queue, finance team investigates

Handling payment processing delays #

Some payments take hours/days (PSP risk review, 3D Secure Authentication). The PSP handles long-running requests:

1. Returns pending status to client → client shows pending + check-status page
2. Notifies via webhook on status change (or payment service polls PSP)

Three-party consistency (client, payment service, PSP) requires idempotency + reconciliation.

Communication among internal services #

• Single receiver: shared message queue: once processed, message removed Figures 7-8: Message queue, single receiver
• Multiple receivers: Kafka: same message consumed by multiple services (payment, analytics, billing) Figure 9: Multiple receivers

For large-scale payment systems with many dependencies → async preferred.

Handling failed payments #

Track payment state: definitive state at any stage, persisted in append-only table.

Figure 10: Handle failed payments

1. Check if failure is retryable → retry queue; non-retryable (invalid input) → error DB
2. Consume retry queue, retry transactions
3. If fails again: retry count < threshold → back to retry queue; ≥ threshold → dead letter queue (debug/inspect)

Exactly-once delivery #

Exactly-once = at-least-once (retry) + at-most-once (idempotency).

Retry (at-least-once) #

Figure 11: Retry

Retry strategies: immediate, fixed intervals, incremental intervals, exponential backoff (1s → 2s → 4s → ...), cancel. Use exponential backoff for persistent network issues. Provide Retry-After header with error codes.

Idempotency (at-most-once) #

Client generates a UUID as idempotency key → HTTP header <idempotency-key: key_value> (Stripe/PayPal recommended).

Scenario 1 (double click): idempotency key = shopping cart ID pre-checkout. Second request seen before → return previous request's status. Concurrent duplicates → 429 Too Many Requests.

Figure 12: Idempotency

Implementation: DB unique key constraint. Insert succeeds → new request. Insert fails (PK exists) → duplicate, reject.

Scenario 2 (PSP processed but response lost): nonce (payment_order_id) → PSP → token. Token uniquely maps to the payment order. Re-send the same token → PSP recognizes the duplicate, returns previous execution status.

Consistency #

Stateful services involved: payment service, ledger, wallet, PSP, DB replicas.

1. Internal services: exactly-once processing
2. Internal ↔ external (PSP): idempotency + reconciliation: never trust an external system fully
3. Replica lag: Option 1: read+write from primary only (wastes replicas). Option 2: consensus algorithms (Paxos, Raft) or consensus-based DBs (YugabyteDB, CockroachDB)

Payment security #

Step 4 of 4: Wrap up #

What we designed:

1. Pay-in / pay-out flows with a 3rd-party PSP (hosted page → no PCI DSS)
2. Reconciliation via nightly settlement files
3. Async communication patterns
4. Failure handling: retry + dead letter queues
5. Exactly-once delivery: retry + idempotency

If you have more time, discuss:

1. Monitoring, alerting, debugging tools
2. Currency exchange, regional payment methods (cash payments in India/Brazil)
3. Google/Apple Pay integration