Metrics Monitoring and Alerting System

Design a scalable metrics monitoring and alerting system for internal use by a large company. Out of scope: log monitoring and distributed tracing.

Figure 1: Popular services

Step 1 of 4: Understand the problem and establish design scope #

Goal: lock in requirements and scale.

Requirements #

1. Monitor operational system metrics: CPU load, memory, disk space, requests/sec, server count. Not business metrics
2. Scale: 100M DAU, 1,000 server pools × 100 machines/pool × 100 metrics/machine → ~10M metrics
3. Data retention policy:
   • Raw data: 7 days
   • 1-minute resolution: 30 days
   • 1-hour resolution: 1 year
4. Alert channels: email, phone, PagerDuty, webhooks

Non-functional:

1. Scalable for growing metrics/alert volume
2. Low query latency for dashboards and alerts
3. Highly reliable (no missed critical alerts)
4. Flexible pipeline for future technology integration

Step 2 of 4: Propose high-level design and get buy-in #

Goal: agree on the 5 components and the storage choice.

Five components #

1. Data collection: gather metrics from sources
2. Data transmission: transfer to the monitoring system
3. Data storage: organize and store incoming data
4. Alert: detect anomalies, generate alerts, send to channels
5. Visualization: present data in graphs/charts

Data model #

Metrics = time series: uniquely identified by metric name + labels (tags).

Format (line protocol):

CPU.load host=webserver01,region=us-west 1613707265 50

Time series structure:

Data access pattern #

1. Write-heavy: ~10M operational metrics written per day at high frequency
2. Read-spiky: visualization + alert queries → bursty read volume

Data storage system #

Do NOT build your own or use a general-purpose DB (MySQL). Relational DBs are not optimized for time-series (complex SQL for moving averages, index per tag needed, poor under constant heavy write).

1. NoSQL (Cassandra, Bigtable): can work but requires deep internal knowledge for a scalable schema
2. Time-series databases (recommended): OpenTSDB (HBase-based → complex), InfluxDB, Prometheus. InfluxDB: 8 cores + 32GB RAM → 250K+ writes/sec. Strong label-based indexing for fast aggregation/analysis. Key: keep labels low cardinality

High-level design #

1. Metrics source: app servers, SQL DBs, message queues
2. Metrics collector: gathers + writes to the time-series DB
3. Time-series database: stores time-series; custom query interface; label indexes
4. Query service: thin wrapper for querying; could be replaced by the DB's own interface
5. Alerting system: sends notifications
6. Visualization system: graphs/charts

Step 3 of 4: Design deep dive #

Goal: collection model, pipeline scaling, storage, alerts, visualization.

Metrics collection #

Occasional data loss is acceptable (counters, CPU). Fire-and-forget OK.

Pull model:

1. Dedicated metric collectors pull from application /metrics endpoints periodically
2. Uses Service Discovery (etcd/Zookeeper) to know endpoints; collectors subscribe to endpoint changes

3. Multiple collectors → must avoid duplicate pulls. Use consistent hashing: each collector is assigned a range; each monitored server maps to exactly one collector

Push model:

1. A collection agent installed on each server → pushes metrics to the collector
2. Can aggregate locally before sending
3. Buffers locally if the collector rejects (risk: data loss if servers are auto-scaled out)
4. Collector cluster with load balancer + auto-scaling

Pull vs Push comparison:

• Pull examples: Prometheus
• Push examples: Amazon CloudWatch, Graphite
• Large orgs likely need both (serverless complicates agent installation)

Scale the metrics transmission pipeline #

Introduce Kafka between the collector and the time-series DB:

1. Highly reliable, scalable message platform
2. Decouples collection from processing
3. Prevents data loss when the DB is unavailable (data retained in Kafka)

Scale through Kafka:

1. Configure partitions by throughput
2. Partition by metric name → consumers can aggregate by metric name
3. Further partition by tags/labels
4. Prioritize important metrics for processing first

Alternative: Facebook's Gorilla: an in-memory TSDB designed to remain highly available for writes without an intermediate queue.

Where aggregations can happen:

1. Collection agent (client-side): simple aggregation (e.g. counter per minute)
2. Ingestion pipeline (pre-write): stream processing (Flink). Reduces write volume significantly. Loses raw data precision. Late-arriving events challenging
3. Query side (post-write): no data loss. Slower (computed at query time against the full dataset)

Query service #

Cluster of query servers between visualization/alert systems and the time-series DB. Decouples clients from storage. Cache layer for query results. Caveat: most industrial visualization/alert systems have strong plugins for popular TSDBs: a query service may be unnecessary.

Time-series database query language #

SQL is painful for time-series (exponential moving average requires nested subqueries). InfluxDB's Flux is simpler:

from(db:"telegraf")
  |> range(start:-1h)
  |> filter(fn: (r) => r._measurement == "foo")
  |> exponentialMovingAverage(size:-10s)

Storage layer #

85% of queries target data from the last 26 hours (Facebook research). A good TSDB harnesses this property.

Data encoding/compression: built into good TSDBs. Store deltas instead of absolute values. Example: 1610087371, 10, 10, 9, 11 (base + deltas) instead of full timestamps.

Downsampling: convert high-res → low-res over time. Example rules:

1. 7 days: no sampling
2. 30 days: 1-minute resolution
3. 1 year: 1-hour resolution

10s → 30s rollup: average 3 consecutive values.

Cold storage: rarely-used inactive data at much lower cost.

Alert system #

1. Alert rules (YAML) loaded into cache
2. Alert manager fetches config from cache
3. Alert manager calls the query service at intervals. If the threshold is violated → creates an alert event
4. Alert store (Cassandra KV) tracks alert states (inactive, pending, firing, resolved). Ensures at-least-once notification
5. Eligible alerts → Kafka
6. Alert consumers pull from Kafka
7. Consumers send notifications (email, SMS, PagerDuty, webhook)

Alert manager also handles: filter/merge/dedupe alerts, access control, retry.

Build vs buy: off-the-shelf alert systems integrate well with popular TSDBs and notification channels. Building your own is hard to justify.

Visualization system #

High-quality visualization is hard to build. Use off-the-shelf (e.g. Grafana): it integrates with many TSDBs.

Final design #

Step 4 of 4: Wrap up #

What we designed:

1. Pull vs push collection models
2. Kafka for scaling the pipeline
3. The right time-series database
4. Downsampling for space efficiency
5. Build vs buy for alerting/visualization: final design integrates all components through Kafka decoupling