c7KE

Design a digital wallet backend supporting cross-wallet balance transfers. Users store money in wallet and spend/transfer later.

Figures 1-2: Digital wallet, Cross-wallet balance transfer

Step 1 - Understand the Problem and Establish Design Scope

Requirements:

• Balance transfer between two digital wallets only
• 1,000,000 TPS (transactions per second)
• Transactional guarantees sufficient
• Reproducibility required: must be able to reconstruct historical balances by replaying data from the beginning (beyond reconciliation — shows discrepancies but not root cause)
• 99.99% availability
• No foreign exchange

Back-of-the-envelope estimation:

• Each transfer = 2 operations (debit + credit) → 2M actual TPS
• Assume 1,000 TPS per DB node → need 2,000 nodes

Goal: maximize per-node TPS to reduce hardware cost.

Step 2 - Propose High-Level Design and Get Buy-In

API Design #

POST /v1/wallet/balance_transfer Params: from_account, to_account, amount (string), currency (ISO 4217), transaction_id (UUID for dedup).

In-memory sharding solution (rejected) #

Figure 3: In-memory solution

Redis cluster, sharded by hash(accountID) % partitionNumber. Zookeeper for sharding config. Stateless wallet service handles transfers by updating two Redis nodes.

Problem: Two updates across nodes are not atomic. Wallet service crash between debit and credit → incomplete transfer. Need atomic transactions.

Distributed transactions #

Database sharding #

Figure 4: Relational database

Replace Redis nodes with transactional relational DB nodes. Still doesn't guarantee atomicity across two different databases.

Two-phase commit (2PC) #

Figure 5: Two-phase commit

1. Coordinator (wallet service) reads/writes on multiple DBs (all locked)
2. Coordinator asks all DBs to prepare
3. All "yes" → commit all. Any "no" → abort all.

Low-level solution: requires special DB transaction modification (X/Open XA standard). Problems: locks held long (low performance), coordinator is single point of failure.

Figure 6: Coordinator crashes

Try-Confirm/Cancel (TC/C) #

Compensating transaction approach. Two phases, each a separate independent transaction (unlike 2PC where both phases are in same transaction).

Example: Transfer $1 from A→C:

Try phase:

Figure 7: Try phase

Wallet service sends: (1) A: start local transaction, reduce balance -$1. (2) C: NOP (always succeeds).

Confirm phase:

Figure 8: Confirm phase

A already updated. C: add $1.

Cancel phase:

Figure 9: Cancel phase

A's Try already committed → start new transaction to add $1 back (reverse). C: NOP.

TC/C = "distributed transaction by compensation." High-level (business logic implements undo). Database-agnostic (any DB with transactions works). Disadvantage: complexity in application layer.

Phase status table #

If wallet service crashes mid-TC/C, recovery needs history. Store progress in transactional DB:

• Distributed transaction ID + content
• Try phase status per DB (not sent / sent / response received)
• Second phase name (Confirm or Cancel)
• Second phase status
• Out-of-order flag

Stored in the DB containing the debit account.

Figure 10: Phase status table

Unbalanced balance #

End of Try phase: A = $0,C=$0 (A had $1,Chad$0 initially). Sum decreased — violates accounting rule. But transactional guarantee maintained. Intermediate inconsistency is normal during distributed transactions.

Figure 11: Unbalanced state

Valid operation orders #

Only choice 1 is valid. Always debit first, NOP on credit in Try phase.

Out-of-order execution #

Network issues → Cancel may arrive before Try. Solution:

• Cancel leaves out-of-order flag: "seen Cancel, not Try yet"
• Try checks flag → if set, return failure

Figure 12: Out-of-order execution

Saga #

De-facto standard in microservices. Linear sequence:

1. Operations ordered. Each independent transaction on own DB.
2. Execute sequentially. When one finishes, next triggered.
3. On failure: rollback from current → first in reverse order using compensating transactions. N operations need 2N operations (N normal + N compensating).

Figure 13: Saga workflow

Coordination:

• Choreography: services subscribe to each other's events (decentralized). Hard to manage with many services.
• Orchestration (preferred): single coordinator instructs services in order.

TC/C vs Saga #

Choice: If no latency requirement or few services → either works (Saga trending in microservices). If latency-sensitive with many services → TC/C (parallel execution).

Event sourcing #

Auditor questions: Do we know balance at any time? How do we know historical/current balances are correct? How to prove correctness after code change?

Event Sourcing (DDD technique) solves all three.

Core concepts #

1. Command: intended action (e.g. "transfer $1 from A to C"). May be invalid. Put in FIFO queue.
2. Event: validated, fulfilled fact (e.g. "transferred $1 from A to C"). Must be deterministic. Uses past tense. One command → 0+ events. Event generation may contain randomness/I/O.
3. State: what changes when Event applied (account balances — map of account→balance).
4. State Machine: (a) validates Commands → generates Events. (b) applies Events → updates State. Must be deterministic (no random I/O or numbers).

Figures 14-15: Static and dynamic views of Event Sourcing

Wallet service example #

Figures 16-17: Command queue, How state machine works

Commands in Kafka. State Machine:

1. Read Command from queue
2. Read balance State from DB
3. Validate. If valid → generate two Events (A:-$1,C:+$1)
4. Read next Event
5. Apply Event: update DB balance

Reproducibility #

All changes saved as immutable history (Event list). DB is just a view of current State. Replay events from beginning → reconstruct any historical state. Immutable Events + deterministic State Machine = identical results every replay.

Figure 18: Reproduce states

CQRS (Command-Query Responsibility Segregation) #

Rather than publishing State, Event Sourcing publishes Events. External world rebuilds custom States.

Figure 19: CQRS architecture

• One write State Machine (validates Commands, generates Events)
• Multiple read-only State Machines build different views (balance query, hourly audit trail, etc.)
• Eventually consistent (lag, but catch up)

Step 3 - Design Deep Dive

High-performance event sourcing #

File-based Command and Event list #

Optimization 1: Save Command/Event to local disk (append-only file) instead of remote Kafka. Sequential writes are fast (faster than random memory access in some cases). Avoids network transit.

Optimization 2: Cache recent Commands/Events in memory. Use mmap — maps disk file to memory array. OS caches file sections in memory. Append-only → nearly all data in memory.

Figure 20: File-based command and event storage

File-based State #

Replace remote relational DB with local file-based store: RocksDB (LSM-based, write-optimized, caches recent data for reads). Or SQLite (file-based relational).

Figure 21: File-based solution

Snapshot #

To accelerate reproducibility: periodically save current State to immutable snapshot file. State Machine resumes from snapshot (not from beginning). Common for finance: snapshot at 00:00 for daily verification. Stored in object storage (HDFS).

Figure 22: Snapshot

Reliable high-performance event sourcing #

Reliability analysis #

4 data types: Command, Event, State, Snapshot.

• State + Snapshot: regenerable from Event list.
• Command: Event generation may contain randomness/I/O → not deterministic → can't guarantee reproducibility of Events from Commands.
• Event: immutable, represents historical facts, can rebuild State. Only Event needs high-reliability guarantee.

Raft consensus #

Replicate Event list across multiple nodes. Requirements: no data loss, same relative order across nodes.

Raft roles: Leader, Candidate, Follower. Leader receives external commands, replicates data. Majority online = system works. 3 nodes tolerate 1 failure, 5 nodes tolerate 2 failures.

Figures 23-24: Raft, Raft node group

Architecture: 3 Event Sourcing nodes, Raft-synchronized Event lists. Leader accepts Commands → converts to Events → appends to local Event list → Raft replicates to followers. All nodes process Event list → update State. Identical Event lists → identical States.

Failure handling:

• Leader crash: Raft elects new leader from healthy nodes. Client resends Command (timeout/error detected).
• Follower crash: retry until restart or replacement.

Distributed event sourcing #

Two limitations:

1. CQRS pull model not real-time.
2. Single Raft group capacity limited → need sharding + distributed transactions.

Pull vs push #

Pull: client periodically polls read-only State Machine. Not real-time, can overload service.

Figure 25: Periodical pulling

Reverse proxy: proxy forwards Command, polls status. Simplifies client, still not real-time.

Figure 26: Pull model with reverse proxy

Push model: read-only State Machine pushes execution status to reverse proxy immediately on receiving Event → feels real-time.

Figure 27: Push model

Distributed transaction (Final design) #

Shard by hash(key) % 2. Use Saga coordinator.

Figure 28: Final design

Happy path (Saga, transfer $1 A→C):

1. User A → Saga coordinator: distributed transaction (A:-$1,C:+$1)
2. Coordinator creates Phase Status record
3. Coordinator sends A:-$1 Command to Partition 1 (account A)
4. Partition 1 Raft leader: stores Command → validates → converts to Event → Raft syncs → applies Event (deduct $1)
5. Partition 1 CQRS syncs data to read path → rebuilds state + execution status
6. Read path pushes status → Saga coordinator 7-8. Coordinator records Partition 1 success in Phase Status table
7. Coordinator sends C:+$1 Command to Partition 2 (account C)
8. Partition 2 Raft leader: stores → validates → converts → syncs → applies Event (add $1)
9. Partition 2 CQRS syncs to read path → rebuilds state + status
10. Read path pushes status → Saga coordinator 13-14. Coordinator records Partition 2 success
11. All operations succeed → distributed transaction complete → respond to caller