Grafana Alerting — Deep Study

1. Overview / TL;DR

Grafana Alerting (a.k.a. "unified alerting", introduced in Grafana 8) is the alerting engine built into Grafana. It lets you define alert rules against any data source (Prometheus, Mimir, Loki, Postgres, etc.), evaluate them on a schedule, and route notifications through contact points — Slack, PagerDuty, email, webhooks — with one consistent rule model and one routing tree. Under the hood it embeds Prometheus Alertmanager, so the routing/silencing/dedup semantics you know from the Prometheus ecosystem apply unchanged.

2. How Grafana Alerting Works

Grafana Alerting is not a single process — it's a pipeline with three stages, each with a clear responsibility. Understanding the stages is what lets you debug "why didn't I get paged":

There are two places a rule can live, and the choice matters for reliability:

3. Anatomy of an Alert Rule

In Grafana's rule builder, a rule is three stacked expressions. In Prometheus (data source–managed) the same thing is written as a single PromQL alerting rule. Here is the mapping:

groups:
- name: service-availability
  rules:
  - alert: HighErrorRate
    expr: |
      sum(rate(http_requests_total{status=~"5.."}[5m]))
        / sum(rate(http_requests_total[5m]))
      > 0.01
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "High 5xx error rate on {{ $labels.service }}"
      description: "{{ $value | humanizePercentage }} errors over the last 5m."

Every field matters:

• expr — a PromQL expression that yields a vector; it's "firing" when it returns at least one element.
• for — the "debounce" duration. Prevents flapping: the condition must hold continuously for this long before paging. 5m is a common default; too short = alert storms, too long = you notice too late.
• labels — attached to the alert, drive routing (e.g. severity=page vs severity=ticket).
• annotations — human-readable text with Go-template variables ({{ $labels.* }}, {{ $value }}) shown in the notification and dashboard.

4. Alert States, Evaluation & Timing

A rule is always in one of a few states, and — crucially — not only "on/off". The two extra states are where most monitoring setups silently fail:

Evaluation interval, group & jitter

• Evaluation interval — how often the rule runs (default 1m). Lower = faster detection, higher = more load on the data source.
• Evaluation group — rules in the same group are evaluated sequentially with the same interval; a slow rule in a group delays its siblings. Keep heavy/slow rules in their own group.
• Jitter — a small random offset Grafana adds so every rule doesn't hit the data source at the exact same instant.
• Pending ≠ silent — the "for" window is where flapping gets absorbed. A 5m "for" on a 1m evaluation interval means ~5 evaluations must all be true before you page.

5. Alerting Philosophy: Symptom vs Cause

The most common reason alerting "doesn't help" is that it fires on the wrong things and trains everyone to ignore it. Google's SRE book distills this into one rule:

Two methods give you a vocabulary of "symptoms" for any service:

6. SLO-Based Burn Rate Alerts

The most reliable way to "prevent downtime" is to alert on how fast you're burning your error budget, not on a single hard threshold. A burn rate is the ratio of your current error rate to your allowed error budget. If you have a 99.9% availability SLO, your error budget is 0.1%, and a burn rate of 1 means you're consuming budget at exactly the rate you can afford over 30 days.

The standard recipe is the multi-window, multi-burn-rate alert (Google SRE): fire only when BOTH a short window shows a fast burn AND a long window confirms it's not a transient blip. For a 99.9% SLO (error budget 0.1%):

groups:
- name: slo-burn-rate
  rules:
  # Error ratio over the last 5 minutes
  - record: job:error_ratio_5m
    expr: |
      sum(rate(http_requests_total{status=~"5.."}[5m]))
        / sum(rate(http_requests_total[5m]))

  # Error ratio over the last 1 hour
  - record: job:error_ratio_1h
    expr: |
      sum(rate(http_requests_total{status=~"5.."}[1h]))
        / sum(rate(http_requests_total[1h]))

  # PAGE: fast burn (5m) AND sustained burn (1h)
  - alert: SLOBurnRateHigh
    expr: |
      job:error_ratio_5m > 14.4 * 0.001
      and
      job:error_ratio_1h > 6 * 0.001
    for: 2m
    labels:
      severity: page
    annotations:
      summary: "Error budget burning too fast ({{ $labels.job }})"
      description: "5m error ratio {{ $value | humanizePercentage }}."

The magic numbers: 14.4× burns the entire 30-day budget in ~1 hour, and 6× over 1h means the incident has already consumed a meaningful slice of budget. The and of two windows is what makes the alert both fast to fire and resistant to flapping — the short window catches the spike, the long window vetoes it if it was a 30-second blip.

7. Unleash-Specific Alert Rules

The specific risk with Unleash is silent staleness. Unleash SDKs cache the last-known flag state locally, so a brief outage usually doesn't crash your services — they keep serving with stale flags. That's a feature (availability), but it means the failure mode is invisible: your app is up, but flag changes stop propagating, and a brand-new instance with an empty cache can't evaluate flags at all. Your alerts must catch that degradation before it becomes a real incident.

7.1 Target down — the foundation

The single most important alert. Prometheus's up metric is 1 when a scrape succeeded, 0 when it didn't. If your Unleash server or Edge stops answering scrapes, everything else is moot:

groups:
- name: unleash-health
  rules:
  - alert: UnleashTargetDown
    expr: up{job=~"unleash-server|unleash-edge"} == 0
    for: 2m
    labels:
      severity: page
    annotations:
      summary: "Unleash target {{ $labels.job }} is not scraping"
      description: "Prometheus cannot reach {{ $labels.instance }}."

7.2 Edge can't reach the Unleash server

Your Unleash Edge proxies flag evaluations to the server. If Edge's upstream connection breaks, Edge falls back to cache — but that cache is now going stale. This is the earliest warning of the "flags stopped updating" failure mode. (Metrics below match the Edge metrics already covered in the Unleash Edge notes.)

  - alert: UnleashEdgeUpstreamErrors
    expr: rate(edge_upstream_errors_total[5m]) > 0
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "Unleash Edge cannot reach the upstream server"
      description: "{{ $value | humanize }} upstream errors/sec over 5m."

7.3 Cache hit rate dropping

A healthy Edge serves most flag evaluations from its cache (high hit rate). A falling hit rate means Edge is hitting the server more — either the cache is being evicted, keys are churning, or upstream is flapping. Sustained low hit rate is a leading indicator of upstream overload:

  - alert: UnleashEdgeLowCacheHitRate
    expr: |
      sum(rate(edge_cache_hit_total[5m]))
        / (sum(rate(edge_cache_hit_total[5m])) + sum(rate(edge_cache_miss_total[5m])))
      < 0.95
    for: 10m
    labels:
      severity: ticket
    annotations:
      summary: "Unleash Edge cache hit rate below 95%"
      description: "Cache hit rate is {{ $value | humanizePercentage }}."

7.4 Flag evaluation latency

If flag lookups get slow, every request that depends on a feature flag gets slow. Edge should answer flag evaluations in single-digit milliseconds; a p99 over ~10ms is a real regression:

  - alert: UnleashEdgeHighLatency
    expr: |
      histogram_quantile(0.99,
        sum(rate(edge_request_duration_seconds_bucket[5m])) by (le))
      > 0.010
    for: 5m
    labels:
      severity: ticket
    annotations:
      summary: "Unleash Edge p99 latency above 10ms"
      description: "p99 is {{ $value | humanizeDuration }}."

7.5 Your apps can't evaluate flags (the one that actually pages)

The real "is my service down?" signal is measured from your app, not from Unleash. Expose (or scrape) the SDK's own error/evaluation counters and alert when flag evaluation starts failing:

  - alert: FeatureFlagEvaluationFailing
    expr: |
      sum(rate(unleash_sdk_evaluation_errors_total[5m])) by (service)
        / sum(rate(unleash_sdk_evaluation_total[5m])) by (service)
      > 0.01
    for: 5m
    labels:
      severity: page
    annotations:
      summary: "{{ $labels.service }} cannot evaluate feature flags"
      description: "{{ $value | humanizePercentage }} of flag evaluations failing."

7.6 The full Unleash alert ladder

8. Notification, Routing & Silencing

Once an alert fires, Grafana's embedded Alertmanager decides who hears about it and how. Two concepts do the heavy lifting:

Silences & mute timings

• Silence — manually suppress alerts matching labels for a fixed window (during a planned maintenance or a known incident). Created in the UI, API, or via amtool.
• Mute timing — a recurring schedule (e.g. "weekends", "after midnight") that routes non-critical alerts to a low-priority channel or drops them entirely.
• Inhibition — one alert suppresses another (e.g. "host down" inhibits all the "service down" alerts on that host), cutting noise at the source.

{{ define "slack.message" }}
*[{{ .Status | toUpper }}]* {{ .CommonLabels.alertname }}
{{ range .Alerts }}
• {{ .Annotations.summary }} ({{ .Labels.service }})
  {{ .Annotations.description }}
{{ end }}
{{ end }}

9. Grafana Alerting vs Alertmanager vs SaaS

In practice these aren't mutually exclusive: the common pattern is Prometheus evaluates the rules (data source–managed), Grafana manages & visualizes them, and PagerDuty/Opsgenie handles on-call escalation as the last-hop contact point. That gets you GitOps-able rules, a nice UI, and real on-call scheduling without betting on one vendor.

10. Best Practices & Pre-Launch Checklist

Before you call your alerting "done", walk this checklist. Every unchecked item is a way to be paged at 3am by something you can't act on — or worse, not paged at all.

References

• 1
  Grafana Alerting — official documentation

  The authoritative reference for alert rules, contact points, notification policies, and the state machine.
• 2
  Alert rule fundamentals — Grafana docs

  The query/reduce/threshold expression model, "for" duration, and evaluation semantics.
• 3
  Prometheus alerting rules — official docs

  The PromQL alerting rule format (expr / for / labels / annotations) used by data source–managed rules.
• 4
  Alerting on SLOs — Google SRE Book

  The canonical treatment of burn rates and the multi-window multi-burn-rate alerting approach.
• 5
  Alerting — Prometheus best practices

  Symptom-vs-cause guidance, alert fatigue, and what to alert on.
• 6
  Unleash technical overview — official docs

  Unleash architecture (server, SDKs, Edge), health/metrics endpoints, and the fail-open cache behavior.
• 7
  Configuring Unleash — metrics & health endpoints

  Prometheus metrics endpoint and health-check configuration for the Unleash server.
• 8
  Unleash Edge — official docs

  Edge metrics (edge_upstream_errors_total, edge_cache_hit_total, edge_request_duration_seconds) used in the Unleash alerts above.