Case 5.3·Production ObservabilitySEV1

“Every dashboard was green for 50 minutes while every customer saw a certificate error.”

case name: Green inside, broken outside

Service: public endpoint · TLS
Impact: 100% of customers blocked by a browser security warning for 50 minutes
Detected by: Customers and social media
Time to resolve: 50 min

Skills you'll use on this case

white-box vs black-box monitoringblackbox_exporterprobe_successprobe_ssl_earliest_cert_expiry

00 It starts

Reproduce this incident in your lab, then work the case alongside the timeline below. Try each query yourself before reading its result.

terminal

$ curl -sI https://expired.badssl.com | head -1

── lab output ──

curl: (60) SSL certificate problem: certificate has expired

badssl.com hosts deliberately broken endpoints: perfect test targets for a probe.

01 The investigation

00:00
FINDING
The certificate on the public load balancer expires at midnight UTC
It had been renewed manually every year. This year, the person who did it had left.
00:05
DEAD END
Every internal signal: healthy
Error ratio: no data (Case 4.4) because requests never REACH ShopLite; browsers stop at the TLS handshake. Latency: fine, for the handful of health checks inside the VPC. Every metric the team had was measured from the inside, and the problem was entirely outside.
00:50
RESOLVED
Certificate renewed manually; ACM with auto-renewal and external probes added the next day

Root cause

A TLS certificate expired. All monitoring was WHITE-BOX, from inside the system, so it could only see requests that reached the application. Nothing tested the service the way a customer does, from outside, through DNS, TLS, and the load balancer.

02 The concepts behind it

White-box and black-box monitoring

WHITE-BOX monitoring uses internal signals (metrics, logs, traces) from the system itself: rich detail, but blind to anything that stops requests from arriving. BLACK-BOX monitoring (probes, synthetic checks) tests from the outside like a user: 'can I resolve the name, complete TLS, get a 200, within 2 seconds?'. It knows little about why, but it's the only thing that sees DNS mistakes, expired certificates, CDN or load-balancer failures, and firewall changes.

blackbox_exporter

A Prometheus exporter that performs probes on demand: HTTP(S), TCP, ICMP, DNS, gRPC. Prometheus 'scrapes' /probe?target=<url>&module=http_2xx, and the exporter returns probe_success, probe_duration_seconds, probe_http_status_code, and TLS details like probe_ssl_earliest_cert_expiry. Run it from where your users are, or at least from outside your VPC.

Expiry is a predictable incident

Certificates, domain registrations, API keys, and cloud credits all expire on a known date. Alert weeks ahead (so there's time to act in working hours), and automate renewal wherever possible: ACM, Let's Encrypt with cert-manager or Caddy.

03 The fix

01Add blackbox_exporter and probe from outside

The relabelling moves the URL from the target list into the target query parameter and points the actual scrape at the exporter. That's the standard blackbox pattern. The lab probes ShopLite internally, a real public site, and a deliberately expired certificate.

obs-lab/prometheus/prometheus.yml + blackbox.ymladd to fileyaml

# compose:
#   blackbox:
#     image: prom/blackbox-exporter:v0.27.0
#     command: --config.file=/etc/blackbox/blackbox.yml
#     volumes: ["./blackbox.yml:/etc/blackbox/blackbox.yml:ro"]

# blackbox.yml
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      preferred_ip_protocol: ip4
      valid_status_codes: [200]

# prometheus.yml — scrape_configs
  - job_name: blackbox
    metrics_path: /probe
    params: { module: [http_2xx] }
    static_configs:
      - targets:
          - http://shoplite:8080/healthz
          - https://grafana.com
          - https://expired.badssl.com
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox:9115

02Read the probe results

PromQL· Prometheus

probe_success
(probe_ssl_earliest_cert_expiry - time()) / 86400

result

probe_success{instance="http://shoplite:8080/healthz"}   1
probe_success{instance="https://grafana.com"}            1
probe_success{instance="https://expired.badssl.com"}     0

{instance="https://grafana.com"}  54.3   (days until certificate expiry)

04 Make sure it never surprises you again

01Alerts: probe failing, certificate expiring

The expiry alert is a ticket at 21 days and should never need to page. The probe alert pages, because it's the closest thing you have to 'customers can't reach us'.

obs-lab/prometheus/alerts.ymladd to fileyaml

  - name: blackbox
    rules:
      - alert: EndpointDown
        expr: probe_success == 0
        for: 2m
        labels: { severity: critical, service: edge }
        annotations:
          summary: "{{ $labels.instance }} is failing external probes"
      - alert: CertificateExpiringSoon
        expr: (probe_ssl_earliest_cert_expiry - time()) / 86400 < 21
        labels: { severity: warning, service: edge }
        annotations:
          summary: "TLS certificate for {{ $labels.instance }} expires in {{ $value | humanize }} days"

05 Your turn: write the query

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.

Which blackbox metric shows how long the TLS handshake took, and how would you graph it per target?

Why should critical probes run from outside your own cloud network?

06 Interview questions from this case

What's the difference between white-box and black-box monitoring, and why do you need both?

0/4 · 0%

Command Palette