“Support has an order ID and a timestamp for a failed checkout. There's no trace for it — we only keep 10%.”

1. 14:00

   REPORT

   Support: 'order failed at 13:52:14 for this customer — what happened?'
2. 14:05

   QUERY

   Search for error traces around that time

   TraceQL· Tempocopy

   { resource.service.name = "shoplite" && span.http.route = "/checkout" && status = error }

   result

   (a handful of results — roughly 1 in 10 of the failed checkouts)

3. 14:07

   FINDING

   Head sampling decided at the START of each request, before anyone knew it would fail

   traceidratio keeps a random 10% of traces, decided when the root span starts. Errors are no more likely to be kept than successes, so 90% of error traces are thrown away, and those are exactly the ones worth keeping. Meanwhile 10% of the millions of boring 200 OK traces are still stored.

4. 16:30

   RESOLVED

   Tail sampling in an OpenTelemetry Collector: keep every error and slow trace, 5% of the rest

1. 01Add an OpenTelemetry Collector with tail sampling

   The contrib distribution includes the tail_sampling processor. Policies are OR-ed: a trace is kept if ANY policy says keep. Point both services' OTEL_EXPORTER_OTLP_ENDPOINT at http://otel-collector:4318, and remove the SDK head-sampling variables so the SDK records everything.

   # compose:
   #   otel-collector:
   #     image: otel/opentelemetry-collector-contrib:0.130.0
   #     command: --config=/etc/otel/config.yml
   #     volumes: ["./otel-collector.yml:/etc/otel/config.yml:ro"]
   
   receivers:
     otlp:
       protocols:
         grpc: { endpoint: 0.0.0.0:4317 }
         http: { endpoint: 0.0.0.0:4318 }
   
   processors:
     tail_sampling:
       decision_wait: 10s
       num_traces: 50000
       policies:
         - name: keep-errors
           type: status_code
           status_code: { status_codes: [ERROR] }
         - name: keep-slow
           type: latency
           latency: { threshold_ms: 1000 }
         - name: baseline
           type: probabilistic
           probabilistic: { sampling_percentage: 5 }
     batch: {}
   
   exporters:
     otlp/tempo:
       endpoint: tempo:4317
       tls: { insecure: true }
   
   service:
     pipelines:
       traces:
         receivers: [otlp]
         processors: [tail_sampling, batch]
         exporters: [otlp/tempo]

2. 02Verify: every failed checkout has a trace

   Compare error checkouts counted by metrics with error traces found by TraceQL for the same 5 minutes. They should now match closely. Then switch the scenario off.

   PromQL· Prometheuscopy

   sum(increase(http_request_duration_seconds_count{job="shoplite", route="/checkout", status="500"}[5m]))

   result

   {}  361   ← and TraceQL now returns ~361 error traces for the same window

   terminalcopy cmd

   $ curl -s -X POST localhost:8080/admin/chaos -H 'content-type: application/json' -d '{"scenario":"errors","enabled":false}'

   ── expected output ──

   {"active":[]}

1. 01Watch the collector itself

   The collector exposes Prometheus metrics on port 8888 by default (add a scrape job). Track spans received versus exported, dropped spans, and memory. A tail-sampling collector that runs out of memory drops traces silently.

   PromQL· Prometheuscopy

   sum(rate(otelcol_receiver_accepted_spans_total[5m])) - sum(rate(otelcol_exporter_sent_spans_total[5m]))

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.