Distributed Tracing in Go: Finding the Needle in the Microservice Haystack
Running a web of Go services taught us that logs alone do not tell the full story. Distributed tracing stitched requests together, let us spot bottlenecks faster, and gave on-call engineers a calmer playbook. This article is the story of why we adopted it - for the setup, propagation, Tempo, and PII mechanics, see Observability in Go: What Goes in a Metric, a Log, and a Trace.
The Debug Session That Broke Our Team
Picture a familiar incident: production alarms fire, dashboards show errors, and the failing request passes through a dozen services with incompatible log formats.
Every team owns its own logging style, timestamps are out of sync, and correlating events means scraping Kibana, Splunk, and raw stdout. Hours disappear while everyone tries to rebuild a single timeline.
That was the moment we decided to stop doing log archaeology and ship distributed tracing.
Once traces were available, incident responders could jump straight to a shared view of the request path. Multi-hour hunts turned into targeted investigations measured in minutes.
What Distributed Tracing Actually Solves
Before tracing, our debugging workflow looked like this:
- Find the failing request ID in the API gateway logs
- Search for that ID in the auth service logs
- Search for that ID in the user service logs
- Search for that ID in the payment service logs
- Repeat across every downstream service
- Try to reconstruct the timeline manually
- Give up and restart everything
After tracing:
- Search for trace ID in Grafana Tempo
- See the entire request flow in one view
- Identify the bottleneck immediately
- Fix the actual problem
The Mechanics: Setup, Propagation, Tempo, PII
This article used to carry its own walkthrough of OpenTelemetry setup, HTTP/gRPC/DB instrumentation, sampling, PII redaction, and testing here. It's gone - not because that content was wrong exactly, but because a newer article on this blog covers the same mechanics with something this one didn't have: a real measured bug. otel.SetTextMapPropagator(...) showed up in this article's setup code with no explanation of what happens if you skip it; the propagation section there shows exactly what happens - three disconnected trace IDs instead of one connected chain, silently, the first time that line was actually missing from a real run - plus the exemplar/cardinality mechanics, a corrected account of how Tempo actually indexes (or doesn't index) span attributes, and PII discipline that applies to logs and traces both.
Keeping here: why tracing was worth adopting in the first place, and what changed once it shipped - the part of this article that was never about a specific SDK version anyway.
Where Tracing Paid Off
With traces available, the same few incident patterns kept showing up—and became much easier to resolve. The three below are composite, illustrative shapes of what recurred, not a log of exact incident timestamps - the resolution times are rough, remembered orders of magnitude, not numbers pulled from a ticketing system:
Case 1: The Mysterious Timeout
- Symptom: random spikes to 30 seconds on the checkout endpoint.
- Trace view: one branch of the request sat idle while the dependency pool filled up.
- Root cause: a rarely hit code path forgot to release database connections.
- Resolution time: about 15 minutes once we pulled the trace, versus multi‑hour log spelunking before.
Case 2: The Performance Regression
- Symptom: p95 latency drifting from ~200 ms to ~2 s after a release.
- Trace view: a group of spans ballooned in the profile service.
- Root cause: an N+1 query introduced in the latest merge.
- Resolution time: under 30 minutes; previously we would bisect builds for half a day.
Case 3: The Ghost Error
- Symptom: a few percent of requests returned 500 with no obvious log trail.
- Trace view: retries cascading between services until a fragile fraud check finally threw.
- Root cause: swallowed exceptions that left spans marked OK while the caller still failed.
- Resolution time: roughly 10 minutes after spotting the trace.
The Hard-Won Lessons
Keeping tracing useful is mostly about people and process:
- Instrumentation must live in the definition of done; otherwise spans drift away from reality.
- Moderate sampling (single-digit percentages for production) balances visibility with cost.
- Traces, metrics, and logs belong in one shared dashboard so incident response starts from the same facts.
- Domain-specific spans—"prepare invoice", "charge card"—make runbooks actionable. Purely technical spans rarely help product teams.
The Bottom Line
Before tracing, root-cause hunts across a dozen services with incompatible log formats routinely ran long, and morale went with them. After rolling it out, the same class of incident got meaningfully faster to close - not because tracing is magic, but because the team could jump straight to a shared timeline instead of reconstructing one by hand from a dozen different logging formats.
P.S.
Distributed tracing did not replace thoughtful logging, but it removed the guesswork. These days we open Grafana Tempo first and only dive into raw logs for supporting detail.