Observability in Go: What Goes in a Metric, a Log, and a Trace

Part 1: The Short Version

Got 6 minutes and want the shape of this without the Prometheus internals? This is the part for you. Part 2 below has the benchmark and the mechanics; this part is the mental model. The whole point of building any of it is a system that surfaces a problem while it's still cheap to fix, instead of one that stays quiet until the problem is already expensive.

Grafana Is a Viewing Layer, Not an Observability System

Say this plainly, because a lot of the mistakes below start from misunderstanding it: for dashboards, Grafana doesn't collect anything, doesn't store anything, and doesn't decide what's worth measuring. It's a window onto whatever Prometheus, Loki, Tempo, Mimir, or something else already collected and stored. (Unified Alerting is the one real exception - it does evaluate alert rules and hold their state itself, rather than just displaying something Prometheus computed. Everything below about dashboards still holds; alerting is its own case, not covered by "just a viewing layer.") "Add it to Grafana" isn't a real instruction for what this article is actually about. The decisions that matter — what to measure, as what kind of signal, at what cardinality — happen in the systems Grafana is mostly just looking at, and they get made badly exactly when someone treats the dashboard tool as if it were the whole observability system.

Three Signals, Three Questions

Metrics, logs, and traces each answer a different question. None of them substitutes for the other two:

• Metrics answer "is something wrong, and roughly what." A dashboard or an alert tells you error rate is up or p99 latency crossed a threshold. It's an aggregate signal, cheap to query, and it isn't about any one request.
• Traces answer "where, for this specific request." Given a trace_id — pulled from an exemplar, a log line, a support ticket — a trace shows which service and which call in the chain was actually slow or failed.
• Logs answer "why." The stack trace, the downstream error message, the validation failure reason that neither a metric nor a trace span carries on its own.

That's also the order an investigation usually follows: a metric says something's wrong, an exemplar on that metric points at a trace, the trace shows where in the request path it happened, and the logs for that trace_id explain why. Wiring trace IDs through all three signals — covered in Part 2's Tempo and exemplars sections — turns that chain into one click at each step instead of a manual correlation exercise across three separate UIs.

What Belongs in a Label - a Quick Checklist

• Bounded by something you control (routes you wrote, a fixed set of statuses) - label it
• Grows with traffic or users (an ID, a raw error string, a timestamp) - don't; use an exemplar (metrics), structured metadata (Loki logs), or just the log line itself
• Personal data in any form - never, in a label or anywhere else in a metric; see PII below for why logs need the same discipline
• Not sure yet - don't add it until you know you need it. A filter expression over existing log content answers most "do we need this as a queryable field" questions before committing to an index cost for it.

Dashboards and Alerts

Collecting metrics without a dashboard anyone actually opens, or an alert that actually fires, is observability theater. The data sits there, but nobody's flying with it until something's already on fire and someone starts building a query from scratch.

Dashboards earn their keep by answering a specific question fast, not by showing everything at once. The RED method (Rate, Errors, Duration - originated by Tom Wilkie, also written up on the now-defunct Weaveworks blog, still live at time of writing) covers what a request-serving service needs on one screen. USE (Utilization, Saturation, Errors) covers a resource — a database, a queue, a disk. Five panels people actually open during every incident beat forty panels nobody scrolls to.

Alerts should be symptom-based rather than cause-based wherever that's possible. "p99 latency on checkout is above 2s for 5 minutes" pages someone; "CPU is at 80% on host-47" usually shouldn't, unless high CPU on that specific host is already known to directly cause a symptom users feel. Cause-based alerts on every resource metric are exactly how alert fatigue happens. A useful gut check before adding any alert: if this fires at 3am, is there a specific action the person paged should take, or does it just tell them something is true that they can't act on until morning anyway? If it's the second one, make it a dashboard panel, not a page.

PII in Logs

The cardinality discipline above has a second reason to apply just as strictly to logs, and it has nothing to do with cost: logs are the most likely place personal data ends up somewhere it shouldn't. An email address, a full name, a raw IP, a phone number dropped into a log line "just for debugging" is a GDPR-relevant data-processing decision, whether or not anyone thought of it that way at the time. Unlike a database row, a log line usually isn't covered by whatever access controls and retention policies the rest of the system has.

• Log identifiers, not the personal data itself. An opaque internal user_id can be useful in a log line, as long as access to those logs stays restricted to whoever actually needs it — not a blanket "any ID is fine" regardless of who can read it, how long it's kept, or what it can be joined against. Under GDPR a pseudonymous internal ID is still personal data, but it carries far less risk than a direct identifier, and it's much easier to answer a "right to erasure" request for, since deleting the mapping from ID to person is enough. A user's email or name usually isn't necessary to debug anything, and that's the data an erasure request would have to physically reach inside the log store.
• Redact at the source, not downstream. A logging middleware that scrubs known-sensitive field names catches this before the data leaves the process. A downstream pipeline that's supposed to redact it later means every log line sat somewhere unredacted first.
• Retention and erasure don't compose for free with immutable log storage. If a deletion request has to remove specific personal data from logs already written, someone has to build that capability. Never putting the personal data in the log in the first place is far cheaper.
• This isn't legal advice. GDPR obligations depend on jurisdiction, data category, and legal basis for processing. What's above is the engineering-practice floor; the actual compliance requirements for a given system are a conversation with whoever owns that at your company.

Fact → Where It Goes

Everything in Part 1, as one lookup table instead of paragraphs - the thing worth actually pasting into a team wiki:

Part 2: The Deep Dive

Everything below is the mechanics behind Part 1's rules: the actual benchmark, and how Prometheus, Loki, and Tempo each handle this differently underneath, or manage to avoid the problem altogether.

What "Cardinality" Actually Means

A Prometheus time series isn't a metric name. It's a metric name plus every unique combination of label values. http_requests_total{route="/orders", method="GET", status="200"} is one series. Add a user_id label and every distinct user who ever hits that route becomes its own series. That's a brand new series, allocated in Prometheus's in-memory head block, not a new value tucked inside an existing one.

In principle that memory cost isn't permanent for any single series: Prometheus marks a series stale once it stops appearing in scrapes for a few minutes, and the next head compaction (every 2 hours by default) evicts stale series from memory, leaving behind a much cheaper compressed chunk on disk for the rest of the retention window. In practice, that relief only arrives for a series that actually stops being scraped - and a CounterVec in client_golang never forgets a label combination once it's been used. Every user_id this demo has ever seen stays in the app's own /metrics output for the life of the process, gets re-scraped forever, and so never goes stale at all.

Checked this properly, not with one before/after pair: reran the load at a realistic 15s scrape interval (2s was overstating how many chunks a real service would generate) and sampled prometheus_tsdb_head_series, process_resident_memory_bytes, go_memstats_heap_inuse_bytes, and prometheus_tsdb_head_chunks every 30 minutes with zero additional traffic. The instance kept running unattended well past the original plan, which turned out to matter - the short window and the long one tell different parts of the same story:

                right after   +31min   +60min   +90min   +120min  +150min   ~22h (12 compactions)
head_series         100,667  100,668  100,668  100,668  100,668  100,714    100,714
RSS (MiB)              256.5    372.0    491.2    450.8    359.8    382.9    434.2
heap_inuse (MiB)       170.7    353.5    414.5    400.2    285.6    259.1    330.3
head_chunks          100,667  100,676  201,393  302,061  402,729  503,443   503,573

Two different things are happening here, and the first pass at this section conflated them. head_series stayed flat throughout, at every horizon - no eviction, confirming the mechanism above. In the first 2.5 hours, head_chunks grew in a straight, unbroken line, roughly +100,000 every 30 minutes, with no compaction visible in the logs - Prometheus's TSDB head doesn't compact on a schedule tied to when data started arriving; it compacts on 2-hour-aligned boundaries relative to the Unix epoch (the actual log lines: a block written at 10:02 UTC, then again at 11:00, 13:00, 15:00... every 2 hours on the hour, twelve times over the following day), and this process happened to start at 07:02, so the first boundary it could hit was nearly 3 hours out - the 2.5-hour window ended just before compaction ever got a chance to run. RSS and heap_inuse didn't wait for that: both peaked around the one-hour mark and dropped by roughly a quarter from ordinary Go garbage collection, even while head_chunks kept climbing without pause - heap_inuse tracking RSS that closely is the tell that this is live heap growth, not an mmap'd, evictable page-cache artifact sitting over a flat heap. Prometheus's head chunks are ordinary Go-heap objects until compaction writes them out, so heap and chunk count move together, and only GC (not compaction) moved memory down inside that first window.

The ~22-hour column is where compaction actually shows up, and it changes the conclusion: twelve compaction cycles ran on schedule, each one logged writing a block and then, one cycle later, deleting the previous block once it aged past the 1-hour retention window - and head_chunks did not keep climbing at +100,000 every 30 minutes for 22 hours straight; it sat at 503,573, within a few hundred of where it was at the 2.5-hour mark, because compaction has been steadily flushing chunks out of the head at roughly the same rate new ones form. RSS and heap_inuse, similarly, sit at 434.2 MiB and 330.3 MiB - inside the same 250-490 MiB band the first 2.5 hours already showed, not higher. So the corrected claim, in full: memory never returns to the pre-load baseline (79.5 MiB idle, ~256 MiB right after 100K users) - that part holds up over 22 hours and twelve compactions, not just a lucky two-point comparison. But it isn't unbounded growth either. Once compaction is running on its normal cadence, the system settles into a persistently elevated steady state - roughly 3-6x the pre-load baseline in this run, oscillating with GC - rather than climbing forever. The first version of this section, measured in a window too short to see a single compaction cycle, made the growth look worse (unbounded) than the fuller picture supports; the corrected finding is narrower but more defensible: this cost is a permanent floor under an idle process, not a peak that fully recedes, and not a runway to infinity either.

So the ongoing cost of an unbounded label isn't just "a growing product mints new values faster than old ones go stale." It's also that a service which never prunes old label combinations - the default, unless you go out of your way to remove them - keeps every one of those series costing memory for as long as the process keeps scraping it, settling into a real, elevated resting cost rather than a one-time spike. A route name is bounded - you wrote the routes, there are a few dozen of them, and they don't grow with traffic. A user ID, an order ID, a session token, a raw error message with an ID baked into the text: none of those are bounded by anything except how many of them your service has ever seen, and unless something actively retires old ones, "how many it has ever seen" only goes up - and once it does, that's the new floor, not a spike that fades.

The Benchmark: What user_id in a Label Really Costs

Setup. A small Go service (net/http + prometheus/client_golang) exposes the same request counter two ways side by side: http_requests_safe_total{route, method, status} (bounded) and http_requests_unsafe_total{route, method, status, user_id} (deliberately not). A local Prometheus 3.0.1 scrapes it, running in Docker with a 2s scrape interval and otherwise default config. A /load?users=N endpoint mints N guaranteed-unique user IDs and increments both counters once per synthetic user, so the experiment controls exactly how many distinct label values get created.

var (
    // Bounded by (route x method x status) - dozens of series, no matter
    // how much traffic or how many users hit the service.
    requestsSafe = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "http_requests_safe_total",
        Help: "Request count, labeled by route/method/status only.",
    }, []string{"route", "method", "status"})

    // user_id added as a label - cardinality now scales with the number
    // of distinct users who have ever made a request, not the number of
    // routes. Every new user is a brand new time series.
    requestsUnsafe = prometheus.NewCounterVec(prometheus.CounterOpts{
        Name: "http_requests_unsafe_total",
        Help: "Same thing, plus user_id - deliberately bad practice, for measurement only.",
    }, []string{"route", "method", "status", "user_id"})
)

Baseline, before any load - this is just Prometheus's own self-scrape overhead:

prometheus_tsdb_head_series:      665
process_resident_memory_bytes:    83,390,464   (79.5 MiB)

After /load?users=50000, one scrape interval later:

prometheus_tsdb_head_series:      50,691   (+50,026)
process_resident_memory_bytes:    164,806,656   (157.2 MiB, +77.6 MiB)
count(http_requests_unsafe_total): 50,000
count(http_requests_safe_total):       2   (unchanged - still just GET+POST x /orders x 200)

After another /load?users=50000 (100,000 distinct users total):

prometheus_tsdb_head_series:      100,691   (+50,000 from the previous step)
process_resident_memory_bytes:    274,051,072   (261.4 MiB, +104.2 MiB from the previous step)
count(http_requests_unsafe_total): 100,000
count(http_requests_safe_total):       2   (still 2)

Overall, 100,026 new series cost 190,660,608 bytes of process-level resident memory: ~1.86 KiB per additional time series, averaged across both load steps (1.59 KiB/series on the first 50K, 2.13 KiB/series on the second 50K). Head-block growth isn't perfectly linear because of map resizing and chunk allocation, but the order of magnitude held both times. That RSS figure is the whole process's memory growth, not an isolated per-series allocation — it includes Prometheus's own heap overhead, goroutine stacks, and mmap'd chunk pages responding to the larger head block, not purely the bytes each series occupies on its own; see the heap-vs-RSS breakdown two sections down for what that overhead actually consists of. The bounded counter didn't move once, at any point, no matter how many users generated traffic through it.

One bookkeeping note before going further: this two-step 50K-then-50K table is its own run, at the original 2s scrape interval. The cardinality-mechanism discussion and the idle-recheck section below it rerun the same experiment from scratch, at a more realistic 15s interval, in a single 100K-user jump instead of two 50K steps - so their series/memory figures (100,667 series, 282.7 MiB post-load) land close to but not identical to this table's (100,691 series, 261.4 MiB). Different sessions, not a live number drifting under you; both are real measurements of the same thing.

Query cost moved too, on a Prometheus with nothing else running and no remote read layer in front of it. Measured by timing /api/v1/query HTTP round trips directly (Python, http.client, one persistent keep-alive connection to avoid paying a fresh TCP/TLS handshake per request, median of 20-25 requests per query) rather than reading Prometheus's own internal stats=all evaluation timings - the number reported is closer to what a dashboard panel or an alert rule actually waits on, not the query engine's isolated CPU time:

sum(rate(http_requests_safe_total[1m]))                          ~0.3ms
sum(rate(http_requests_unsafe_total[1m])) by (user_id)            ~258ms
sum(rate(http_requests_unsafe_total[1m]))                          ~196ms

Sit with that third row for a second. It runs the exact same query as the row above it, minus by (user_id), so it produces exactly the aggregate the safe query does: one number, not 100,000. It still costs ~196ms, not ~0.3ms — about three-quarters of the grouped query's cost, consistent across repeated runs (72-80%) even though the two absolute numbers moved between runs (150-260ms range) with nothing else changing on an idle machine. Dropping by (user_id) barely helps, because grouping was never where the cost lived. Both queries have to scan all 100,000 series to compute a rate over them. The query planner can't know in advance that only 2 of them matter; from where it stands, all 100,000 are equally real series it has to touch. Whether the output has one row or 100,000 changes the serialization cost at the margin. It doesn't change the scan. That's the real mechanism behind an unbounded label's query cost: every query over that metric pays for every series it ever created, whether or not that particular query even mentions user_id, and that cost grows with the product rather than with what any single query asks for. This is also one idle instance answering one query at a time, with no remote-read layer or concurrent load in front of it. A production Prometheus runs many more metrics, queried by many more dashboards and alert rules at once, so whatever a label like this costs here gets paid many times over there.

The full setup — Go service, docker-compose Prometheus config, the exact curl/PromQL commands used to produce every number above — is in the gist. It's reproducible in about five minutes if you want to check it on your own machine, or push it further than 100K series.

Guardrails, and Finding a Violator That's Already There

Everything above measures what an unbounded label costs once it exists. Two separate questions worth asking before that: how do you stop your own scrape target from doing this in the first place, and how do you find out whether one already is, in a Prometheus you didn't build from scratch?

Three scrape_config options exist specifically to cap this at ingestion, all defaulting to 0 (unlimited) so they have to be turned on deliberately: sample_limit caps samples accepted per scrape, label_limit caps labels per sample, label_value_length_limit caps the byte length of any one label value. All three fail the entire scrape if exceeded, not just the offending series - a blunt instrument on purpose, closer to a circuit breaker than a filter, so a target that starts minting unbounded labels goes to zero data instead of slowly poisoning the head block. metric_relabel_configs with action: drop is the finer-grained tool: it can drop specific label/metric combinations by regex before they ever reach the head, useful for the case where you know exactly which label is the problem and want to keep the rest of that metric.

For a Prometheus that's already running, the /status/tsdb page in its own web UI is the fastest way to check: it lists the top metric names by series count, the top labels by number of distinct values, and the top series by memory. No PromQL required. The equivalent as a query, useful for alerting on this or checking from outside the UI, sweeps every series and counts them by metric name:

topk(10, count by (__name__)({__name__=~".+"}))

That query itself isn't cheap - it touches every series in the head to build the count - so it's a diagnostic to run occasionally, not something to put in a dashboard that refreshes every 15 seconds.

Once high-cardinality series already exist and can't be un-added without breaking whatever depends on them, recording rules are the mitigation this article's own benchmark points straight at: a rule that precomputes sum(rate(http_requests_unsafe_total[5m])) on a schedule moves that 100,000-series scan from every dashboard load and every alert evaluation to one scheduled write, at the cost of one more series (the recorded one) and however stale the precomputed value is between evaluations. It doesn't reduce the underlying cardinality - the raw series and their memory cost are still there - but it stops every consumer of that aggregate from paying the query's full scan cost independently, which is exactly the multiplication problem the benchmark above ends on.

The Fix: Exemplars, Not Labels

The actual use case behind "let's add user_id to the metric" is almost always "when this is slow or failing, I want to find the specific request to look at." That's a real need. A label is just the wrong tool for it, and the wrong data shape too, since a label defines what a metric's aggregate values mean rather than pointing at one specific occurrence.

Exemplars are the tool actually built for this: a small, bounded piece of extra data, typically a trace ID, attached to one observation inside a histogram or counter rather than to the series identity itself. A histogram has a fixed number of buckets. Exemplars ride along with individual observations that land in those buckets, so the bucket count stays what it's always been while still giving you a way to jump from "this bucket had a slow observation" to the exact trace behind it. The code below shows the histogram case, but a plain counter gets the same treatment through ExemplarAdder - counter.(prometheus.ExemplarAdder).AddWithExemplar(1, labels) instead of a plain .Inc(), same idea, one line different.

import (
    "context"

    "github.com/prometheus/client_golang/prometheus"
    "go.opentelemetry.io/otel/trace"
)

var requestDuration = prometheus.NewHistogramVec(prometheus.HistogramOpts{
    Name:    "http_request_duration_seconds",
    Help:    "Request duration, bounded labels only.",
    Buckets: prometheus.DefBuckets,
}, []string{"route", "method", "status"})

func observeWithExemplar(ctx context.Context, route, method, status string, seconds float64) {
    observer := requestDuration.WithLabelValues(route, method, status)

    // The trace ID comes from the same ctx every handler already has -
    // nothing to thread through signatures by hand (see the propagation
    // section below). No span in the context, no exemplar: plain observe.
    sc := trace.SpanContextFromContext(ctx)
    if !sc.IsValid() {
        observer.Observe(seconds)
        return
    }

    // The Histogram implementation also implements ExemplarObserver -
    // this type assertion is documented as safe, not a hack.
    observer.(prometheus.ExemplarObserver).ObserveWithExemplar(
        seconds,
        prometheus.Labels{"trace_id": sc.TraceID().String()},
    )
}

Two things this needs that a plain /metrics endpoint doesn't provide by default. Your app has to expose OpenMetrics format instead of the older Prometheus text format, because exemplars aren't representable in the old one. And Prometheus itself needs --enable-feature=exemplar-storage, off by default because it allocates a fixed-size in-memory circular buffer for exemplars — one buffer for the whole instance, sized by --storage.exemplars.exemplars-limit (100,000 by default), not one per series. Old exemplars just get overwritten, so the cost stays bounded regardless of series count. promhttp.HandlerFor with EnableOpenMetrics: true covers the Go side of the format switch.

The result in Grafana is a histogram panel with little diamond markers on individual buckets, each one clickable straight into the matching trace in Tempo. No label ever carries the trace ID or the user ID. The series stays exactly as bounded as http_requests_safe_total above, and the "find this specific request" need gets served by a mechanism built to carry exactly one bounded reference per observation instead of one unbounded label per user.

Propagating trace_id Across Functions and Services

Everything above assumes a trace_id is just there, ready to attach to an exemplar or a log line. It isn't, on its own, unless something actually carries it from the first function that handles a request through every downstream call, including across an HTTP or gRPC boundary to a completely different service. That's what makes a trace worth more than a single span: a request that touches five services shows up as one trace with five spans, all sharing the same trace_id, ordered and timed against each other, instead of five unrelated logs someone has to correlate by timestamp and hope they got the right five.

In Go, that propagation rides on context.Context, the same value already threaded through every function that takes one for cancellation, which is exactly why it works for this too. Within a process, nothing needs passing explicitly. A span stored in the context via trace.ContextWithSpan — or, more commonly, started with tracer.Start(ctx, "name"), which does the same thing and returns a new context — is visible to every function downstream that has that ctx, all the way down the call stack, with no traceID string parameter to thread through signatures by hand.

Across a process boundary, the context itself can't cross the wire. What crosses is an HTTP header (traceparent, the W3C Trace Context standard) or a gRPC metadata entry carrying the same trace and span IDs, which the receiving service turns back into a context value on its side. otelhttp.NewTransport injects that header on the way out; otelhttp.NewHandler extracts it on the way in. Wrap both ends and propagation happens automatically. Miss the one step that's easy to miss, and every hop silently starts a new, disconnected trace instead of continuing the caller's — which is exactly what happened running this locally before it got added:

otel.SetTracerProvider(tp)

// Without this, otelhttp falls back to a no-op propagator: the client
// never injects a traceparent header, the server never extracts one,
// and every hop silently starts its own new trace instead of
// continuing the caller's.
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
    propagation.TraceContext{},
    propagation.Baggage{},
))

// Outgoing: injects traceparent into every request this client sends.
client := &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}

// Incoming: extracts traceparent (if present) and starts a child span
// under it instead of a fresh root trace.
handler := otelhttp.NewHandler(myHandler, "orders.create")

That comment describes a real bug, not a hypothetical one. The first run of this exact code, without the SetTextMapPropagator line, produced a different trace_id at every hop: three independent traces instead of one connected chain, silently, with no error. Adding that one call fixed it:

missing propagator - three unrelated traces:
orders:    trace_id=34cf5ebec99fff8bfc227992fcb726d6
inventory: trace_id=1e9c7750c6fe8efc5865e9354a48e477

propagator set - one trace, two spans:
orders:    trace_id=458b592cb534aef26c3d61cdad28a096  span_id=05df14001f8c3d59
inventory: trace_id=458b592cb534aef26c3d61cdad28a096  span_id=4df2bffa8ed59b3b

context.Context is the most convenient way to carry a trace_id there is, precisely because nothing has to be built for it. It's already the value every function that might need cancellation or a deadline takes as its first argument, so a trace riding along in the same context costs nothing extra to thread through a call stack. Extracting it for a log line, once the context actually carries one, works the same way regardless of which function or service it happens in. The slog.Handler shown below adds it to every log record automatically, so no call site has to remember to do it:

type traceIDHandler struct{ slog.Handler }

func (h traceIDHandler) Handle(ctx context.Context, r slog.Record) error {
    if span := trace.SpanContextFromContext(ctx); span.IsValid() {
        r.AddAttrs(
            slog.String("trace_id", span.TraceID().String()),
            slog.String("span_id", span.SpanID().String()),
        )
    }
    return h.Handler.Handle(ctx, r)
}

This is the mechanism behind Part 1's "connecting the three signals" idea, made concrete: the same trace_id shows up in the exemplar on a slow metric, on every log line for that request across every service it touched, and as the thread tying together every span in the trace itself, because one context.Context, propagated correctly, is the only thing that has to carry it. Same ID in three places, and none of them a label with unbounded cardinality. Nobody's giving up on finding the specific request; the pointer to it just lives somewhere built to hold one, instead of somewhere that counts every value as a permanent new thing to track. Full code — a three-hop local chain, stdout trace export, the bug and the fix both included — is in the gist. For what changed operationally after actually rolling tracing out across a real set of services, see Distributed Tracing in Go.

Three Decisions That Are Actually Go-Specific

Most of what's above applies just as directly to a Python or Java service instrumented the same way - the cardinality mechanics don't care what emitted the label. Three choices in this stack are genuinely Go decisions, with Go consequences:

client_golang vs. the OpenTelemetry Go metrics SDK. Every code sample in this article uses client_golang directly, which is the older, still dominant choice for a Prometheus-shaped service - but OpenTelemetry's own Go SDK can produce metrics too, exported over OTLP instead of scraped as /metrics. The two aren't interchangeable in what they make easy: client_golang's exemplar support (used throughout this article) is mature and directly tied to Prometheus's own storage; OTel's metrics SDK is the natural choice if traces and metrics are already going out over OTLP to the same collector, and it's what produces the exponential histograms mentioned in the histogram section, which client_golang's native histograms don't do. Picking one is a real decision with a real blast radius - once a service has hundreds of metrics, migrating the instrumentation layer isn't a weekend project - and "which SDK" isn't a question this article's cardinality rules depend on either way.

What runtime/metrics is worth actually watching. collectors.NewGoCollector() registers Go-runtime metrics for free - GC pauses, heap size, goroutine count, all pulled from runtime/metrics under the hood - and it's easy to wire up and then never look at again. The ones worth an actual dashboard panel, not just existing: go_goroutines (a steady climb with no plateau is a leak, exactly the shape this blog's goroutine-leak article is built around), go_gc_duration_seconds (GC pause time - matters directly for p99 latency on a service where that's tracked), and go_memstats_heap_inuse_bytes next to process_resident_memory_bytes, for the same reason it mattered in this article's own benchmark: RSS alone conflates live heap with everything else the OS happens to be holding for the process, and the gap between the two is itself informative, not noise to average away.

Tuning BatchSpanProcessor. The propagation example above starts a tracer provider without touching its defaults, which is fine for a demo and usually wrong to leave alone in production: WithMaxQueueSize (2048 by default) is how many spans can queue before older ones start getting dropped under a traffic spike, WithBatchTimeout (5s) trades export latency against fewer, bigger batches, and WithMaxExportBatchSize (512) caps how much goes out per batch once the timeout or the queue fills first. A service producing spans faster than the default queue drains loses traces silently, and "silently" is literal here: the Go SDK's BatchSpanProcessor doesn't emit a dropped-span counter or log a warning when the queue is full, it just drops them (a known, open gap - tracked upstream). There's nothing built-in to alert on here yet; the honest mitigation today is sizing the queue generously for your actual span volume and treating trace gaps under load as a real possibility, not something a missing-data alert would ever catch.

Loki Has the Same Trap, Different Mechanism

One flag before this section and the next: everything about Prometheus and OpenMetrics above is a vendor-neutral standard, CNCF-graduated, implemented the same way regardless of who's running it. Loki and Tempo aren't that. They're Grafana Labs' own products, one architectural choice among several for logs and traces, not a universal property of how logs or traces work in general. Elasticsearch/OpenSearch, the other common choice for logs, indexes everything by default and has a completely different cardinality story than Loki's stream model, so the "put it in a log instead" mistake looks different there. Jaeger's storage backends aren't built the way Tempo's object-storage design is either. What follows describes Loki and Tempo specifically, because they pair naturally with Prometheus and Grafana and that's the stack this article set out to cover, not because every log or trace backend behaves the same way underneath.

Moving "put it in a log instead" doesn't dodge the problem. Loki has its own version of the same mistake: mechanically different, same underlying error. Loki indexes by label, the same word Prometheus uses, but what blows up is different. Each unique label combination becomes a separate stream, and Loki has to maintain an index entry and chunk boundaries per stream. Per Grafana's own documentation, high cardinality "causes Loki to create many streams... which causes Loki to build a huge index and flush thousands of tiny chunks to the object store." Loki defaults to a limit of 15 index labels per stream specifically because of this. It's configurable (max_label_names_per_series), but the default exists as a guardrail against exactly this mistake, not an arbitrary number.

The fix has the same shape as the Prometheus one, moving the unbounded value out of the indexed dimension, but the mechanism differs: Loki has a dedicated feature for exactly this, called structured metadata. It carries high-cardinality fields like a user ID or order ID alongside the log line without making them part of the stream identity, so you can still filter and search on them (| user_id = "u_12345" in LogQL) without every distinct value creating a new stream. The pattern to actually apply: put things that describe where the log came from — service, environment, a handful of route prefixes — in labels; put things that describe this specific event — user ID, order ID, request ID, the actual error message — in structured metadata or the log line itself, queried with a filter expression, not a label matcher.

Why Tempo Mostly Doesn't Have This Problem

Tempo's design sidesteps this category of problem almost entirely. Here's the actual mechanism, rather than just taking that on faith. Per Tempo's own architecture docs, it "writes traces directly to object storage... skipping the indexing layer entirely." What's actually in object storage is columnar Parquet blocks (the vParquet formats, since Tempo 2.0): span attributes live in columns, and a lookup is either "fetch this trace ID" or a scan of the relevant columns over a time range, not an inverted index over every attribute value the way a search engine, or Loki's stream index, maintains. A trace with a user_id attribute on every span costs Tempo nothing extra in index terms, because there's no per-attribute index to grow in the first place. The value just becomes one more entry in a column.

That's also the tradeoff to know about before assuming Tempo makes tags free everywhere. With no inverted index, "show me every trace where user_id was X" is a column scan, not an index lookup. TraceQL handles this reasonably well — { span.user_id = "u_12345" } over the last few hours is a normal query, and frequently-searched attributes can be promoted to dedicated Parquet columns to make it faster — but the work still scales with the amount of trace data in the time range, and it's bounded by a query time window and a max search duration rather than a full-retention search. Tempo doesn't punish you for adding user_id the way Prometheus does. It also doesn't reward you with a cheap "find all traces for this user" either. The practical pattern is still the exemplar one above: metrics and logs carry the trace ID as the pointer, and Tempo is where you land once you already know which trace you're looking for.

Classic vs Native Histograms: One More Cardinality Multiplier

Labels aren't the only thing that multiplies series count. A classic Prometheus histogram is stored as separate _bucket/_sum/_count series, one per bucket boundary. The http_request_duration_seconds histogram from the exemplar section uses prometheus.DefBuckets: 11 boundaries, plus the mandatory +Inf bucket, so 12 _bucket series, plus _sum and _count — 14 series for every route/method/status combination. Fifty combinations that would be 50 series on a counter become 700 on a histogram. That's the verbose representation, and it sits on top of whatever the labels already cost. A native histogram (Prometheus's term) or an exponential histogram (OpenTelemetry's) is one series carrying all its buckets, with bucket boundaries derived from a scale factor instead of a hand-written list. That same 50-combination histogram stays at 50 series, and gets better resolution in the tails as a bonus. Both need turning on: NativeHistogramBucketFactor on the Go side, --enable-feature=native-histograms on the Prometheus side, and in Mimir, native-histogram ingestion enabled for the tenant. None of this is a drop-in default anywhere. Classic vs native is the decision to make on purpose, and the sizing math — buckets-as-series vs one series — is worth doing before a histogram with many label combinations goes live, especially once cardinality is already the thing you're trying to control.

This usually surfaces as a question about transport: should you send to Mimir over remote_write or OTLP? The two get conflated, so it's worth pulling them apart. Grafana, as Part 1 said, stores nothing; the thing you're shipping to is Mimir, or Grafana Cloud, which is Mimir underneath. Prometheus's own remote_write is snappy-compressed protobuf over HTTP, and it's what a Prometheus server speaks natively — point it at a Mimir endpoint and you're done, no extra component. OTLP goes through the OpenTelemetry Collector, and per Grafana's own Mimir documentation, it's now the recommended path even for Prometheus-shaped metrics. Neither transport decides the histogram question for you, though: remote_write has carried native histograms since Prometheus 2.40, and OTLP will happily carry a classic explicit-bucket histogram too. The one thing OTLP adds is that OpenTelemetry's exponential histograms keep their explicit min/max on the way in, which Prometheus's native format doesn't have. Pick the transport for operational reasons — what's already running, what else it needs to carry — and pick classic vs native for the series count.

Pitfalls That Aren't About Cardinality

Cardinality is the headline mistake here, but a few others show up just as often:

• Scrape interval vs retention cost is a real tradeoff, not just a config default. Scraping every 15s — the interval most example configs ship with, though Prometheus's own default is 1m — for a metric nobody queries more granularly than "last hour" is pure storage cost with no benefit. Scrape interval is set per job, not per metric, so lowering it for slow-moving metrics like queue depth (not request latency) means exposing them on a separate endpoint or target with its own job. It's a bit of plumbing, but a legitimate way to cut cost without touching cardinality at all.
• Alerting on a series that can simply stop existing. An alert rule over a label combination that's actually rare, like one specific status code on one specific route, evaluates to an empty result the moment that combination hasn't been seen recently. An empty result doesn't fire; it just goes quiet, and nobody notices until the outage it was supposed to catch. Guard the ones that matter with absent() or or vector(0) so "no data" is a state the rule knows about instead of one it silently treats as fine.
• Status codes and error messages as raw strings in labels. "500: connection refused to 10.2.3.4:5432" as a label value is the same mistake as user_id, just less obviously so — it looks like a small fixed set of "the errors we get" until the IP address or the port number starts varying.
• Sampling head-first when tail-based would answer the actual question. Head-based sampling, deciding to keep a trace before you know the outcome, is cheap and is what most default OpenTelemetry SDK configs do. But it means the trace for the one request that actually failed has the same chance of being dropped as any boring successful one. Tail-based sampling, deciding after seeing the outcome at the collector, costs more infrastructure but actually guarantees the failed requests are the ones you kept.

What's Measured vs What's Documented

• ~1.86 KiB resident memory per extra Prometheus series from a user_id label. Measured for this article, real Prometheus 3.0.1, real docker-compose setup, reproducible from the gist.
• Bounded counter stays flat regardless of load volume. Measured, same setup. It stayed at 2 series through 100,000 synthetic users.
• Series don't get evicted and memory settles into a persistently elevated steady state, not a full return to baseline - but it's a floor, not unbounded growth, once compaction actually runs. Measured, same setup, sampled every 30 minutes for 2.5 hours, then again after ~22 hours and twelve on-schedule compactions: prometheus_tsdb_head_series flat throughout at every horizon; prometheus_tsdb_head_chunks climbed in an unbroken line for the first 2.5 hours (too short a window to see a single 2-hour-epoch-aligned compaction), then held roughly flat (503,573 vs 503,443) across the next 22 hours and twelve compaction cycles logged writing and deleting blocks on schedule; go_memstats_heap_inuse_bytes tracked process_resident_memory_bytes closely at every point, both oscillating with GC inside a 250-490 MiB band rather than climbing monotonically or dropping back to the ~79.5 MiB idle baseline - confirming the elevated cost is real live heap that settles into a steady state, not an mmap page-cache artifact and not runaway growth either.
• Aggregating away the high-cardinality label barely helps. Dropping by (user_id) cut about a quarter of the cost, not most of it (72-80% of the grouped query's cost remained, across repeated runs). Measured, same setup. Absolute query times moved between runs on the same idle machine (150-230ms), so trust the ratio between the two queries here rather than either one's ms in isolation. Neither is a production number, and none of this was re-tested at higher series counts or under concurrent query load.
• Exemplars need OpenMetrics format plus --enable-feature=exemplar-storage; the code above produces a real exemplar line. Verified against the demo service directly — curl .../metrics/openmetrics shows a trace_id exemplar attached to a bucket line, series count unaffected. The Prometheus-side flag requirement itself is documented, not re-tested against a scraping Prometheus here.
• Trace propagation across an HTTP boundary via otelhttp, and the no-propagator bug. Measured for this article: the demo three-hop chain, run repeatedly, both broken (no propagator) and fixed.
• Loki's 15-label default and structured metadata as the fix. Documented (Grafana's own Loki docs), not independently benchmarked here.
• Tempo's no-index, object-storage, trace-ID-lookup design. Documented (Grafana's own Tempo architecture docs), not independently benchmarked here.
• Classic histogram series math (DefBuckets = 14 series per label combination), native-histogram ingestion needing to be enabled in Mimir, and OTLP as Grafana's recommended path in. Documented (Grafana's own Mimir docs), not independently benchmarked here.

The cardinality numbers and the exemplar output are the parts of this article that were actually run against real software instead of repeated from a vendor's page. Everything else is accurately sourced but not independently re-verified the way the benchmark tables in this blog's other performance articles are. It's worth knowing which parts of an observability article to trust as "someone ran this" versus "someone read the docs correctly" — this one is honest about being both.

This Isn't a Set-and-Forget System

Everything above describes a snapshot, not a settled state, and it's worth being precise about what actually goes stale, because "observability decays over time" isn't quite right either. Code that genuinely doesn't change keeps whatever metrics, logs, and traces were built for it valid indefinitely. Nothing about the calendar alone breaks a dashboard. What breaks it is the codebase moving without the observability around it moving the same amount: a new route that never gets added to the dashboard, a label value that used to be bounded — a plan tier, a feature flag — picking up a value nobody planned capacity for, a metric that kept its name through a refactor but no longer measures what the dashboard's title still claims it does. The maintenance burden here tracks how much the system underneath a signal keeps changing, not how much time has passed since someone last looked at it. That's exactly why the label/log/trace decisions in this article aren't a checklist to run once at launch. They're a discipline to keep applying every time the code they're watching changes, for as long as it keeps changing.