Command Palette

Search for a command to run...

Hectal

Observability · an incident casebook

Every lesson is an outage.
You're on call.

You don't learn observability from a list of tools. You learn it at 2 a.m., when the dashboard says everything is fine and customers say it isn't. This course is 24 incidents like that. Each one runs in a lab on your laptop: ShopLite plus a full Prometheus, Grafana, Loki, Tempo, and OpenTelemetry stack. You trigger the failure, investigate it with real queries, find the root cause, and then fix both the bug and the monitoring that missed it.

Open Case 0.1 See all cases
6 chapters24 incidents96 queries to run

How every case works

  1. 00

    The page

    An alert fires or a customer complains. You get exactly what the on-call engineer got.

  2. 01

    The investigation

    A timestamped timeline of real PromQL, LogQL, and TraceQL queries, with their results, charts, traces, and the dead ends.

  3. 02

    The concepts

    Only after you've felt the problem: why histograms, why labels, why sampling. The theory, explained properly.

  4. 03

    Fix and prevent

    Fix the bug, then fix the monitoring, so the next occurrence pages the right person in minutes.

  5. 04

    Your turn

    Write the queries yourself against your own lab, then compare with the answer and with other learners.

The lab you'll run

Prometheus

metrics + alerts

Grafana · Loki · Tempo

dashboards, logs, traces

OpenTelemetry

instrumentation

k6

steady traffic

Postgres

ShopLite's database

Chapters

CH 0

Seeing Anything at All

SEV2SEV3SEV2SEV2

Four incidents that teach the three signals: metrics tell you THAT something is wrong, logs tell you WHAT, traces tell you WHERE.

CH 1

Metrics in Anger

SEV3SEV1SEV2SEV2

Four incidents where the metrics were there, but reading them wrong, or collecting them wrong, made things worse.

CH 2

Logs at Scale

SEV3SEV1SEV2SEV3

Four incidents about logs as a system in their own right: their volume, what they must never contain, how to line them up with change, and how to parse the ones you didn't write.

CH 3

Tracing Deep Dive

SEV3SEV1SEV3SEV3

Four incidents that only make sense as a trace: death by a thousand queries, retries that amplify an outage, a trace that silently breaks at a queue, and the trace you needed but didn't keep.

CH 4

Alerts & SLOs

SEV2LABSEV2SEV1

Four incidents about the alerting itself: pages nobody should get, reliability nobody could agree on, thresholds that are both too noisy and too slow, and the alert that couldn't fire.

CH 5

Production Observability

SEV2LABSEV1SEV3

Four incidents about the observability platform itself: the monitor that silently stopped monitoring, the data you needed but no longer had, the outage only an outsider could see, and the dashboard nobody could find.