Design a Notification System

Three notification types: mobile push notification, SMS, email.

Step 1 of 4: Understand the problem and establish design scope #

Goal: lock in requirements and volume.

Requirements:

1. Supports push notification, SMS, and email
2. Soft real-time: fast delivery, slight delay acceptable under high load
3. Supported devices: iOS, Android, laptop/desktop
4. Notifications triggered by client apps or scheduled server-side
5. Users can opt out
6. Daily volume: 10M mobile push, 1M SMS, 5M emails

Step 2 of 4: Propose high-level design and get buy-in #

Goal: understand the delivery channels, then fix the initial design's SPOF with queues.

Notification types #

iOS Push: provider → APNS (Apple Push Notification Service) → iOS device. Requires device token + payload (JSON).

Android Push: similar flow using Firebase Cloud Messaging (FCM).

SMS: third-party services like Twilio, Nexmo.

Email: commercial services like Sendgrid, Mailchimp.

Contact info gathering flow #

When a user installs/signs up, API servers collect contact info. DB schema:

1. user table: email, phone number
2. device table: device tokens (one user → multiple devices)

Initial notification sending flow #

1. Services 1–N: microservices, cron jobs, etc. that trigger notifications
2. Notification system: centerpiece; provides APIs, builds payloads for third-party services
3. Third-party services: deliver notifications. Must be extensible (FCM unavailable in China → alternatives like Jpush, PushY)

Problems with the initial design: SPOF (single server), hard to scale independently, performance bottleneck.

Improved design #

Changes:

1. DB and cache moved out of the notification server
2. Multiple notification servers with horizontal scaling
3. Message queues decouple components: each notification type gets its own queue (outage in one third-party service doesn't affect others)

Components (left to right):

1. Services 1–N: send notifications via APIs
2. Notification servers: validate requests, fetch metadata from cache/DB, push events to message queues
3. Cache: user info, device info, notification templates
4. DB: user, notification, settings data
5. Message queues: separate queue per notification type (iOS PN, Android PN, SMS, Email)
6. Workers: pull events from queues, send to third-party services

API example: POST https://api.example.com/v/sms/send

Flow:

1. Service calls API → 2. Notification server fetches metadata → 3. Event pushed to queue → 4. Workers pull and send → 5. Third-party delivers → 6. User receives

Step 3 of 4: Design deep dive #

Goal: reliability, dedupe, and the supporting components.

Reliability #

Prevent data loss: notification data persisted in a notification log DB + retry mechanism.

Exactly-once delivery? No: distributed systems can produce duplicates. Use dedupe via event ID: if the event ID was seen before, discard. (See ref [5] for why exactly-once is impossible.)

Additional components #

1. Notification templates: preformatted notifications with customizable parameters (item name, date, CTA). Benefits: consistent format, fewer errors, saves time
2. Notification settings: users control opt-in per channel (user_id, channel, opt_in). Check before sending
3. Rate limiting: cap frequency to prevent overwhelming users (excessive notifications → users disable them entirely)
4. Retry mechanism: failed delivery → re-enqueue. Persistent failure → alert developers
5. Security: appKey/appSecret for push notification APIs. Only verified clients can send
6. Monitor queued notifications: track total queued count. If large → add more workers

Events tracking: track open rate, click rate, engagement via analytics service integration.

Updated design #

Additions: authentication + rate-limiting on notification servers, retry mechanism, notification templates, monitoring and tracking systems.

Step 4 of 4: Wrap up #

Key design elements:

1. Message queues for decoupling
2. Robust retry mechanism
3. Security via appKey/appSecret
4. Tracking/monitoring at every stage
5. User opt-out respect + rate limiting