“Some receipt emails arrive 20 minutes late. Every checkout trace ends at 'insert into jobs' — and then nothing.”

1. 09:00

   FINDING

   How receipts work: checkout inserts a job row; a worker loop picks jobs up

   It's a Postgres-backed job queue: checkout writes insert into jobs (kind, payload), and a worker polls with for update skip locked. It's a common, solid pattern, and completely invisible to tracing unless you carry the context yourself.

   // in /checkout, after the order insert:
   await db.query("insert into jobs (kind, payload) values ('receipt', $1)", [{ productId, qty }]);
   
   // worker, started once at boot:
   setInterval(async () => {
     const { rows } = await db.query(
       "delete from jobs where id = (select id from jobs order by id limit 1 for update skip locked) returning *");
     if (rows[0]) await sendReceipt(rows[0].payload);
   }, 500);
   
   // db/init.sql:  create table jobs (id bigserial primary key, kind text, payload jsonb,
   //                                   created_at timestamptz default now());

2. 09:20

   QUERY

   Search for the receipt work in Tempo

   The worker's database queries show up, but each one is the ROOT of its own tiny trace: no parent, no link to the checkout that created the job. The setInterval callback runs outside any request, so there's no active context to inherit.

   TraceQL· Tempocopy

   { resource.service.name = "shoplite" && name =~ "pg.query:DELETE.*" }

   result

   Trace ID          Root span                     Spans  Duration
   91ab…             pg.query:DELETE shoplite      1      2ms
   77c0…             pg.query:DELETE shoplite      1      2ms
   ...  (thousands of 1-span orphan traces)

3. 09:40

   HYPOTHESIS

   Late receipts = time waiting in the queue, but we can't measure it per order

   The interesting latency lives BETWEEN the two halves: from insert to pickup. With the trace broken at the queue, there's no way to see it for any specific order.
4. 11:00

   RESOLVED

   Context carried through the job row; one trace per order, queue time visible

1. 01Inject into the job, extract in the worker

   Store the carrier in a trace jsonb column. The worker restores the context and runs its work inside a new span whose PARENT is the checkout. It also records how long the job waited, the number that actually explains late receipts.

   const { context, propagation, trace } = require("@opentelemetry/api");
   const tracer = trace.getTracer("shoplite");
   
   // producer — in /checkout
   const carrier = {};
   propagation.inject(context.active(), carrier);          // { traceparent: "00-…-…-01" }
   await db.query("insert into jobs (kind, payload, trace) values ('receipt', $1, $2)", [{ productId, qty }, carrier]);
   
   // consumer — the worker loop
   const job = rows[0];
   const parent = propagation.extract(context.active(), job.trace ?? {});
   await context.with(parent, () =>
     tracer.startActiveSpan("send-receipt", async (span) => {
       span.setAttribute("queue.wait_ms", Date.now() - new Date(job.created_at).getTime());
       try {
         await sendReceipt(job.payload);
       } finally {
         span.end();
       }
     }),
   );
   
   // db/init.sql:  alter table jobs add column trace jsonb;

1. 01Query queue wait across all orders

   With the attribute on the span, TraceQL finds slow-queued receipts directly, each one linked to its full checkout trace. With span metrics on, send-receipt also gets its own latency histogram.

   TraceQL· Tempocopy

   { name = "send-receipt" && span.queue.wait_ms > 60000 }

Run each one against your lab before revealing the answer. Share your own version in the comments; there's usually more than one correct query.