Monitoring System at Scale — Deep Study

1. Overview / TL;DR

This is a worked system-design answer to "design a monitoring system". The premise that makes it interesting: a monitoring system looks like just "collect metrics, draw graphs, fire alerts" — the difficulty is doing all three while hundreds of thousands of hosts stream data in continuously, queries reach back weeks, alerts must fire in seconds, and the system has to keep working when part of it breaks.

2. Requirements & Scale Math

The post anchors everything to one number: 500,000 servers, each reporting ~100 metrics every 10 seconds.

500,000 hosts × 100 metrics / 10 s
  = 500,000 × 10 samples per second
  = 5,000,000 samples/sec peak

Five million samples a second is the number that kills the naive design. The "simplest possible chain" — Agent → Ingestion Service → Time-Series DB — fails because (a) a single synchronous write path can't absorb bursts, (b) any storage hiccup back-pressures the agents and you lose data, and (c) you have no replay buffer when a shard needs maintenance. That's the whole justification for inserting Kafka.

What the post doesn't quantify (my own back-of-envelope)

Worth doing because it's usually the follow-up question. Assuming good TSDB compression (~1–2 bytes per sample after Gorilla-style delta encoding), 5M samples/sec works out to roughly 0.6–0.9 TB of raw samples per day, before index overhead — on the order of 200–300 TB/year if you kept raw resolution for a full year, which is exactly why retention tiers and downsampling aren't optional.

On the wire it's worse before it's better: a text-format sample with a dozen labels is easily 200–500 bytes, so raw ingest bandwidth sits around 1–2 GB/s (~10 Gbit/s). Collectors therefore batch and compress client-side, and Kafka topic layout must be sharded finely enough that one partition never becomes the bottleneck.

3. Write Path — Kafka as the Shock Absorber

The naive chain gets one insert: a durable log between collection and storage.

Server Agent        push, local buffer, drop-by-policy
      │
      ▼
Ingestion Service   validate, shard by metric, per-tenant quota
      │
      ▼
Kafka               buffer, replay, ordered per partition
      │
      ▼
Consumer group      batch write, backpressure-aware
      │
      ▼
Time-Series DB      compressed, time-indexed, rollups on write

• Peak shaving — Kafka holds the burst. Every 10 seconds the fleet sends a synchronized wave; consumers drain it at storage's sustainable rate.
• Decoupling — the ingestion service doesn't know or care whether the TSDB is healthy. Storage can be restarted, re-sharded, or rolled without agents noticing.
• Durability + replay — if a consumer or storage shard dies, the data is still in the log. Fixed retention window means you can replay the last N hours after a bad deploy.
• Horizontal scale — more partitions and more consumers means more write throughput, linearly, until storage itself is the limit.

4. Storage — Why a Time-Series DB, Not a Normal One

Metrics are append-only, timestamped, usually monotonically increasing in time, and queried as ranges of "series × time window". A general-purpose relational or document database is the wrong shape: it pays for random-access updates the workload never does, and its index blows up on the label sets that make metrics useful.

Retention tiers

The post's rollup scheme doubles as a retention policy. Raw high-resolution points are expensive per byte and rarely queried after a week; aggregated points are cheap and answer most long-range questions.

5. Query Path — Rollups + Redis

Scanning weeks of raw samples to draw a 30-day graph is the read-side version of the naive write path. The fix is to pre-aggregate on write, not to optimize the scan.

TSDB ──▶ Query Service ──▶ Redis ──▶ Dashboard
(raw +     (pick the coarsest      (short TTL,
 rollups)    rollup that still        per-query key)
             answers the range)

6. Alerting — Two Paths, Two Latency Budgets

The post's most useful structural insight: not all alerts deserve the same machinery, so the alert path forks.

                 ┌── A. rule-based ─▶ Alert Service ─▶ query TSDB ─▶ Notifier
Kafka ───────────┤                     (threshold / SLO evaluation, periodic)
 (same stream)   │
                 └── B. streaming ──▶ Flink ─▶ window agg + rules ─▶ Notifier
                                       (seconds or less, never touches storage)

• Notification fan-out — triggered alerts route to chat (Slack), paging (PagerDuty-style), SMS, or webhooks, with grouping, deduplication, silencing, and escalation handled by the notifier, not by each rule.
• Alert on missing data too — a host that stops reporting is the most important alert of all, and neither path finds it by comparing values. You need explicit staleness/absence rules.
• State consistency — running rules in two engines means two sources of truth. Duplicate firing is the classic failure: the same condition alerts from both paths, paging twice. Dedup keys and one owning engine per rule solve it.

7. High Cardinality — The Trap Nobody Sees Coming

The post calls this the most easily overlooked problem, and it's the one that actually kills monitoring platforms in production. A series is a unique combination of metric name + label values. Labels are meant to be bounded: a service name, a route, a status code. Put an unbounded value in a label and every single request mints a new series that must be indexed, stored, and scanned forever.

bad:   http_requests_total{user_id="8f3a…", request_id="b41c…"}   → 1 series per request
       (500k hosts × 100 metrics of this shape = index explosion)

good:  http_requests_total{service="checkout", route="/pay", status="500"}
       (bounded: services × routes × status codes)

• Cost is superlinear — storage, index size, query planning, and memory all grow with distinct series, and a single runaway label can multiply the working set of the whole cluster.
• It's a client-side mistake — by the time the TSDB sees it, the damage is in the index. Hence the post's fix: a Cardinality Tracker in Redis on the ingest path that counts distinct values per label (approximate counts via HyperLogLog are enough) and rejects, samples, or re-labels before admitting data to the pipeline.
• Blast radius control — per-tenant quotas and "quarantine this label" policies mean one team's instrumentation bug degrades one team's dashboards, not the fleet's.

8. Reliability & Graceful Degradation

The system must survive its own failures — and note the irony that monitoring is often blamed during incidents precisely because nobody designed its degradation path.

Degradation ordering is a deliberate decision, not an accident: when the pipeline is saturated, drop debug and per-route metrics, keep RED metrics, and keep the paging path alive. Also worth stating out loud in an interview: queue depth is a first-class alert — if Kafka lag grows without bound, the system is silently losing data even though every process is "up".

9. Interview Framing — How to Actually Deliver This

The post's closing point is the real lesson: what separates candidates is not knowing that you need a queue, it's being able to walk from "it collects metrics" through massive writes, long-range queries, real-time alerts, and failure recovery — in that order, with reasons.

Likely follow-ups and short answers

10. Gaps, Caveats & What the Post Leaves Out

Read the source as an interview sketch, not a reference architecture. Being explicit about its limits is more useful than treating it as complete.

Missing from the post

• Collection layer details — service discovery, scrape/report scheduling, and the agent's own local buffer + retry behaviour when the ingest gateway is unreachable.
• Transport format — OpenMetrics / Prometheus remote_write / OTLP / proprietary binary, and whether compression is on the wire.
• Kafka specifics — partition count and keying strategy, replication factor, retention window, delivery semantics, and how consumer lag is monitored.
• Rollup correctness — which functions are mergeable (count/sum/min/max vs average and percentiles), and how late-arriving data is handled back into already-computed rollups.
• Query language & cost controls — PromQL/LogQL-class expression layer, query timeouts and caps, and how a single expensive dashboard query is prevented from starving others.
• Alerting lifecycle — grouping, inhibition, silencing, runbook links, escalation policy, and on-call ergonomics; also SLO/burn-rate semantics rather than raw thresholds.
• Cost model — per-series or per-sample pricing, storage class tiering, and the actual decision of what to stop collecting.
• Build vs buy — nothing in the post addresses whether to operate this at all instead of buying a managed service.

Where I'd push back

• Two alert paths is real complexity. Running both a rule engine against storage and a Flink pipeline is two systems to operate, two ways to be wrong, and a dedup problem — justified only if you genuinely have sub-second paging requirements.
• Redis as cardinality tracker needs a failure story. If the tracker goes down, do you fail open (admit everything, risk an index explosion) or fail closed (reject data)? Either answer is defensible; silence isn't.
• Kafka is not free. The post treats the queue as pure upside; in practice it adds operational surface, latency, ordering caveats, and a new failure domain that can lose data if under-provisioned. Pull-based Prometheus-scale designs skip it entirely and reach comparable scale with remote-write sharding.

11. This Design vs OSS Stack vs Managed SaaS

The post describes one point in a design space. The two neighbours are worth knowing, because "which of these would you actually build?" is the follow-up that separates memorisation from judgement.

References

• 1
  【系统设计】监控系统 — Xiaohongshu post (硅谷打工仔)

  Primary source for sections 1–8: the 10-slide interview walkthrough of this architecture. Chinese; the final architecture slide is the author-verified one.
• 2
  Apache Kafka — Documentation

  Partitions, replication, consumer groups, and delivery semantics — the properties the write path depends on.
• 3
  Prometheus — Metric and Label Naming Best Practices

  The canonical guidance behind section 7: labels must come from a bounded set; no user IDs, request IDs, or timestamps as labels.
• 4
  Prometheus — Storage

  How a production TSDB lays out blocks, WAL, and retention — a concrete instance of the generic "time-series database" in section 4.
• 5
  Gorilla: A Fast, Scalable, In-Memory Time Series Database (VLDB 2015)

  Origin of the delta-of-delta + XOR compression scheme that underpins the ~1–2 bytes per sample estimate in section 2.
• 6
  Thanos — Highly Available Prometheus Setup with Long-Term Storage

  The scale-out alternative: sidecar + object storage, global query layer, downsampling — no Kafka in the path.
• 7
  Grafana Mimir — Horizontally Scalable Long-Term Storage for Prometheus

  Multi-tenant, sharded TSDB with per-tenant limits — the practical implementation of the sharding + quota ideas in sections 7–8.
• 8
  Apache Flink — Documentation

  Event-time windows, watermarks, and state — the semantics the streaming alert path (section 6, path B) relies on.
• 9
  VictoriaMetrics — Documentation

  A widely deployed single-binary TSDB emphasising compression and low cardinality cost; useful contrast to multi-component designs.
• 10
  Google SRE Book — Monitoring Distributed Systems

  Why alerts should fire on symptoms and SLO burn rather than every cause — the framing added in sections 6 and 9.