API Governance — Keeping Every API Working

1. Overview / TL;DR

As a fullstack team lead you own N projects, each exposing a handful of APIs — and "is everything still working?" becomes an unanswerable question. The moment you rely on someone manually curling endpoints before a release, things break silently in between. The fix is not one tool, it's a layered defense model: each layer catches a different failure at a different time.

Related deep-dives in this study repo: Apache APISIX (gateway layer), Grafana Alerting (L5 rules & SLOs), Observability Stack (metrics/logs/traces), and Sentry (error monitoring).

2. Why Manual Checks Fail

"Just test the APIs before release" works for one project and collapses at five. The failure modes are structural, not about discipline:

3. The 5-Layer Defense Model

Each layer answers one question, and together they close the full lifecycle: did we break it (L1), is it alive (L2), can users reach it (L3), do we know what it is (L4), did anyone get told (L5).

• 1
  Contract testing (L1) — automated schema/behavior checks in CI that fail the build when a change breaks a consumer. Cheapest fix point: the bug never deploys.
• 2
  Health & readiness endpoints (L2) — every service exposes /healthz and /readyz; the orchestrator (K8s, Docker, PM2) and load balancers probe them constantly and restart/route around failures.
• 3
  Synthetic monitoring (L3) — external probes hit your real public endpoints from outside the network, every 1–5 minutes, and flag downtime, latency spikes, and broken critical flows before users complain.
• 4
  Gateway & registry (L4) — route all traffic through one gateway (APISIX/Kong/Traefik) with automatic upstream health checks, plus an OpenAPI registry so every endpoint and its owner is discoverable.
• 5
  Alerting & dashboards (L5) — one "API health" dashboard per project with SLOs, and alerts that route to the owning team in seconds.

4. Contract Testing (L1) — Break It Before It Deploys

A contract is a machine-checkable description of what an API promises: endpoints, methods, request/response schemas, status codes. When either side changes, tests verify the change still satisfies the contract — in CI, so the build fails instead of the production call.

Consumer-driven contracts (Pact)

The most powerful variant for multi-project teams: each consumer (frontend, BFF, other service) records the exact requests it makes and the responses it expects. The provider replays those expectations in its CI. If the provider would break a consumer, its pipeline fails before deploy — even if the provider team never met the consumer team.

# .github/workflows/api-tests.yml (per project)
- uses: actions/checkout@v4
- run: npx @redocly/cli lint openapi.yaml          # schema standards
- run: npx @redocly/cli bundle openapi.yaml -o bundled.yaml
- run: npx @redocly/cli build-docs openapi.yaml -o docs.html
- run: npx newman run api-tests.postman_collection.json \
    --env-var baseUrl=${{ vars.API_BASE_URL }} \
    --reporters junit --reporter-junit-export results.xml
- uses: actions/upload-artifact@v4                    # results into CI UI
  with: { name: newman-results, path: results.xml }

5. Health & Readiness Endpoints (L2) — Runtime Truth

Every service must expose two endpoints. Orchestrators and gateways probe them, which turns "is the API working?" from a manual question into an automated one.

GET /readyz
200 OK
{
  "status": "ready",
  "version": "2.4.1",            // which build is actually serving
  "checks": {
    "postgres":   { "ok": true,  "latency_ms": 3 },
    "redis":      { "ok": true,  "latency_ms": 1 },
    "auth-upstream": { "ok": false, "error": "connect timeout" }
  },
  "uptime_seconds": 482123
}

livenessProbe:
  httpGet: { path: /healthz, port: 8080 }
  initialDelaySeconds: 10
  periodSeconds: 10
readinessProbe:
  httpGet: { path: /readyz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 3

6. Synthetic Monitoring (L3) — Eyes From Outside

Health endpoints are inside your network; they can't tell you what a real user experiences. Synthetic checks run from external vantage points against your public endpoints, every minute, around the clock — including on the DNS, TLS, and gateway path that internal probes skip.

# prometheus.yml
scrape_configs:
  - job_name: blackbox
    metrics_path: /probe
    params: { module: [http_2xx] }
    static_configs:
      - targets:
          - https://api.project-a.com/healthz
          - https://api.project-b.com/v1/ping
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance

7. Gateway & Registry (L4) — Know What Exists

You can't govern APIs you can't enumerate. Two moves turn the sprawl into a manageable list: funnel traffic through a gateway, and publish every contract to a registry.

One gateway, not many entry points

A gateway (Apache APISIX, Kong, Traefik) becomes the single front door: every route is declared in one place, and upstream health checks are built in — APISIX removes a dead upstream from the load-balancer pool automatically and can alert on it. See our APISIX deep-dive for the full architecture.

8. Alerting & Dashboards (L5) — Know the Moment It Breaks

Monitoring data nobody reads is just expensive storage. The last layer converts metrics into a single "who owns it, what's broken" answer. If you already run Grafana + Prometheus, see our Grafana Alerting deep-dive; error-level detail is in the Sentry note.

groups:
  - name: api-slo
    rules:
      - alert: API Availability SLO
        expr: |
          (1 - (
            sum(rate(probe_success == 0[5m])) by (instance)
            / sum(rate(probe_success[5m])) by (instance)
          )) * 100 < 99.9
        for: 10m
        labels: { severity: critical, team: "platform" }
        annotations:
          summary: "{{ $labels.instance }} below 99.9% availability"

9. The Team-Lead Rollout Playbook (2 Weeks)

You don't need a platform team or a budget line — you need a spreadsheet and two weeks of incremental automation. Each step is independently valuable, so partial progress still pays off.

• 1
  Day 1 — Inventory. Spreadsheet: project, base URL, envs (staging/prod), endpoints, auth, owner. This single artifact is the governance backbone — everything else references it.
• 2
  Days 2–3 — Ship health endpoints. Add /healthz + /readyz to every service (FastAPI/NestJS/Spring make this a one-liner). Wire K8s probes or Docker healthchecks. Now the orchestrator restarts dead services automatically.
• 3
  Day 4 — Expose OpenAPI. Generate specs from code (no hand-written docs), commit them per repo, and start a CI lint step (Redocly/Spectral) that fails on breaking changes.
• 4
  Days 5–6 — Contract tests in CI. Add a Newman or Pact step to each project's pipeline. From now on, a breaking change is caught at merge time, not at 2am in prod.
• 5
  Days 7–8 — Synthetic checks. Point UptimeRobot (or Blackbox + Grafana) at every public endpoint in the inventory. Add the team Slack channel as the alert destination. Watch the "is X down?" pings stop.
• 6
  Days 9–10 — Dashboard & SLOs. One Grafana panel per project: availability, p95, error rate. Add the 99.9% burn-rate alert. Pin dashboards in team channels.
• 7
  Day 11 — Gateway consolidation (optional phase 2). Route new endpoints through APISIX/Kong with upstream health checks; migrate existing ones incrementally. This is the highest-effort layer — schedule it, don't block on it.
• 8
  Day 12 — Weekly 30-minute API review. Walk the dashboard with the team: what broke, what's slow, what needs a deprecation date. Governance is a rhythm, not a one-time project.

10. Tooling Comparison

11. Decision Guide — Start Where You Are

References

• 1
  Pact — Consumer-Driven Contract Testing

  The standard for cross-team contract tests: consumers record expectations, providers verify them in CI.
• 2
  OpenAPI Initiative

  The OAS spec — the single source of truth for endpoint schemas, and the backbone of any API registry.
• 3
  Schemathesis — Property-Based API Testing

  Generates edge-case requests from OpenAPI specs and finds spec violations automatically.
• 4
  Prometheus Blackbox Exporter

  Self-hosted synthetic probes (HTTP/TCP/ICMP/TLS) exposed as Prometheus metrics.
• 5
  Grafana Synthetic Monitoring

  Hosted probes plus k6 scripted multi-step checks with built-in alerting and dashboards.
• 6
  Kubernetes Liveness, Readiness & Startup Probes

  Official guide to health endpoints, probe semantics, and the restart-vs-routing behavior.
• 7
  Apache APISIX — Upstream Health Checks

  How a gateway removes dead upstreams from the pool automatically — L4 in practice.
• 8
  Backstage — Developer Portal

  Spotify's open platform for cataloging services/APIs and their owners in one place.
• 9
  Newman — Postman Collections in CI

  Runs Postman collections headlessly in pipelines; the fastest way to get API tests running in CI.
• 10
  Stoplight — API Design & Registry

  OpenAPI design, linting, and a hosted API catalog for team-wide discovery.