Command Palette

Search for a command to run...

Hectal
← All projects

Project 10 of 12 · project brief

Full Observability: OpenTelemetry, Prometheus, Grafana, Loki, Tempo

Instrument the Spring Boot services with OpenTelemetry and build metrics, logs, and traces that connect, plus SLOs and alerting that pages the right person.

Advanced 4 days 4 milestones

The scenario

Last week checkout was slow for 40 minutes and it took the team an hour to find which service was at fault. Your brief: make every request traceable across services, correlate logs with traces, define SLOs for checkout, and alert on them, so the next incident is diagnosed in minutes.

Before you start

Stack

OpenTelemetry Java agentOpenTelemetry CollectorPrometheus (kube-prometheus-stack)GrafanaLoki + Fluent Bit/AlloyTempoAlertmanagerSloth/Pyrra

Target architecture

Project 10 architecturediagram
Rendering diagram…

Deliverables and requirements

You will hand in

  • OTel auto-instrumentation for all Java services via the OTel Operator or agent
  • Collector pipeline (receivers, processors like batch and tail sampling, exporters)
  • Grafana dashboards per service (RED) with links from metrics → traces → logs
  • SLOs for checkout availability and latency with multi-window burn-rate alerts and runbooks

Functional

  • Any slow checkout request can be followed across services in a trace
  • From a trace, jump to that request's logs (shared trace_id)
  • An SLO burn alert fires within minutes of a real degradation

Non-functional

  • Metric label cardinality controlled (no user IDs in labels)
  • Logs are structured JSON with PII masked
  • Retention defined per signal; the monitoring stack itself is monitored

Milestones

  1. 1

    Metrics foundation

    Done when: kube-prometheus-stack running; services scraped; RED dashboards.

    • Install kube-prometheus-stack via Argo CD
    • Expose Micrometer Prometheus metrics; ServiceMonitors for each service
    • Dashboard: request rate, error rate, p50/p95/p99 duration per endpoint

    Prove it works

    terminal
    $ curl -s 'localhost:9090/api/v1/query?query=sum(rate(http_server_requests_seconds_count[5m]))by(service)' | jq -r '.data.result[] | "\(.metric.service) \(.value[1])"'
    ── expected output ──
    api 212.4
    checkout 48.7
    catalog 164.1
  2. 2

    Traces with OpenTelemetry

    Done when: Distributed traces across services in Tempo.

    • Deploy the OTel Collector (and Operator for auto-instrumentation injection)
    • Annotate Deployments to inject the Java agent; set OTEL_SERVICE_NAME and resource attributes
    • Tail sampling: keep 100% of errors and slow traces, 10% of the rest

    Prove it works

    terminal
    $ # Grafana Explore → Tempo: { resource.service.name = "checkout" && duration > 1s }
    ── expected output ──
    Trace 4bf92f3577b34da6: checkout POST /orders 1.84s
    ├─ catalog GET /products/42 12ms
    ├─ payments POST /charge 1.61s ← slow
    └─ kafka produce orders 4ms
  3. 3

    Logs correlated with traces

    Done when: JSON logs in Loki containing trace IDs, linked both ways in Grafana.

    • Logback JSON encoder with trace_id/span_id from MDC
    • Collect with Alloy or Fluent Bit to Loki; label only by namespace/app
    • Grafana data source links: Tempo → Loki (by trace_id), Loki → Tempo (derived field)

    Prove it works

    terminal
    $ # Grafana Explore → Loki
    ── expected output ──
    {app="checkout"} | json | trace_id="4bf92f3577b34da6"
    → 7 log lines, 'payments timeout after 1500ms, retrying (1/2)'
  4. 4

    SLOs, alerts, and runbooks

    Done when: Checkout SLOs with burn-rate alerts routed to on-call with a runbook link.

    • Define SLOs (99.5% success, 95% of requests < 500 ms over 28 days) with Sloth or Pyrra
    • Multi-window, multi-burn-rate alerts; page on fast burn, ticket on slow
    • Alertmanager routing by team; every alert has a runbook URL annotation

    Prove it works

    terminal
    $ amtool alert query --alertmanager.url=http://localhost:9093 slo=checkout-availability
    ── expected output ──
    Alertname Starts At Summary
    CheckoutAvailabilityBurnFast 2026-09-27 14:02:11 IST Error budget burning 14x (1h/5m windows)

Would you run this in production?

  • ☐Cardinality budget and limits on the collector/Prometheus
  • ☐Sampling strategy keeps all errors and slow traces
  • ☐PII masked in logs and span attributes
  • ☐Alerts are symptom-based (SLOs), each with a runbook
  • ☐Meta-monitoring: alerts if Prometheus, Loki, or the collector are down

Stretch goals

  • Long-term metrics with Thanos or Mimir
  • Continuous profiling with Pyroscope
  • Synthetic checks from outside (Blackbox exporter) for the checkout journey

Show it off

Résumé bullet

Built end-to-end observability for a microservices platform with OpenTelemetry, Prometheus, Loki, Tempo, and Grafana; correlated metrics, logs, and traces and implemented SLO burn-rate alerting, cutting incident diagnosis from ~1 hour to minutes.

Demo script

  • Inject latency (Chaos Mesh) into payments, then follow alert → dashboard → exemplar → trace → logs
  • Show the SLO dashboard's error budget
  • Show the tail-sampling policy keeping every error trace

Interview questions about this project

01

Metrics, logs, and traces: when do you use each?

02

What is a burn-rate alert?