Observability for Microservices: Traces, Metrics, and Logs That Answer Questions

In a monolith, debugging a slow request means reading one log file and maybe attaching a profiler. In a microservice architecture, one click can fan out to a dozen services, three databases, and a message broker. When that request takes four seconds, the question becomes: where did the time go?

Monitoring tells you that something is wrong. Observability is the ability to ask why, including questions you didn't anticipate, using the data your system already produces. That data comes in three complementary forms.

The three signals and what each is good at

The value comes from linking them: an alert fires from a metric, you jump to example traces from that time window, and from a slow span you jump to the logs for that exact request.

Distributed tracing

A trace represents one request's path through the system. It's made of spans, where each span is a timed operation (an HTTP handler, a DB query, a cache lookup) with a parent. Visualized as a waterfall, a trace shows immediately which hop was slow or failed.

Context propagation

For spans from different services to join one trace, each service must pass trace context along. The W3C traceparent header is the standard:

traceparent: 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01

Instrumentation libraries inject this header on outbound calls and extract it on inbound ones. For message brokers, put it in message headers so asynchronous hops stay connected.

Instrumenting with OpenTelemetry

OpenTelemetry (OTel) is the vendor-neutral standard for traces, metrics, and logs. Instrument once and send the data to any backend. In Go:

func (h *Handler) Checkout(w http.ResponseWriter, r *http.Request) {
    ctx, span := tracer.Start(r.Context(), "checkout")
    defer span.End()

    span.SetAttributes(
        attribute.String("cart.id", cartID),
        attribute.Int("cart.items", len(items)),
    )

    if err := h.payments.Charge(ctx, total); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "payment failed")
        http.Error(w, "payment failed", http.StatusBadGateway)
        return
    }
}

Most of the value comes from auto-instrumentation: middleware for HTTP servers and clients, gRPC interceptors, and database driver wrappers. Add manual spans only around meaningful business operations.

Sampling

Storing every span from a high-traffic system is expensive. Common strategies:

• Head sampling: decide at the start of a request, for example keeping 5%. Cheap, but you may miss the rare slow request.
• Tail sampling: buffer complete traces in a collector and keep the interesting ones, such as all errors, anything over 1s, plus a small random share of normal traffic. More work to run, but far more useful.

Metrics: measure what users feel

Metrics are cheap to store and fast to query, which makes them the right basis for dashboards and alerts. Two frameworks help you pick what to measure.

RED for request-driven services

• Rate: requests per second
• Errors: failed requests per second
• Duration: latency distribution (histograms, not averages)

USE for resources (CPU, pools, queues)

• Utilization: how busy it is
• Saturation: how much work is waiting
• Errors: failure count

Latency needs percentiles

An average hides pain. If p50 is 80ms and p99 is 3s, 1 in 100 users has a terrible experience, and a user who makes 50 requests per session will almost certainly hit it. Record latency as a histogram so you can compute p50, p95, and p99 at query time.

Watch cardinality

Every unique combination of label values creates a new time series. Labels like method and status_code are fine. Labels like user_id or raw URL paths with IDs can create millions of series and bring down your metrics backend. Normalize paths to route templates (/orders/:id) before recording them.

Structured logs

Plain-text logs are for humans reading one line at a time. Structured logs, meaning key-value pairs usually encoded as JSON, are for machines searching millions of lines:

logger.InfoContext(ctx, "payment captured",
    slog.String("order_id", orderID),
    slog.Int64("amount_cents", amount),
    slog.String("provider", "stripe"),
    slog.Duration("latency", elapsed),
)

{"time":"2026-09-24T10:12:03Z","level":"INFO","msg":"payment captured",
 "order_id":"ord_8812","amount_cents":4599,"provider":"stripe",
 "latency":"212ms","trace_id":"4bf92f3577b34da6a3ce929d0e0e4736"}

Practices that make logs useful:

• Include trace_id and span_id in every log line. That single field connects logs to traces.
• Use consistent field names across services. If one service logs userId and another logs user_id, cross-service queries get painful.
• Log events, not narration. "Payment captured" with fields beats five lines of "entering function…".
• Never log secrets or personal data. Redact tokens, passwords, and card numbers at the logger level.
• Choose levels deliberately. ERROR should mean someone may need to act. If errors are logged for expected conditions like 404s, people learn to ignore the level.

SLOs: turning signals into decisions

A Service Level Objective states what "good enough" means from the user's point of view, for example "99.5% of checkout requests succeed in under 800ms over 28 days."

The remaining 0.5% is your error budget. It changes the conversation:

• Budget left? Ship features and take reasonable risks.
• Budget burning fast? Slow down and prioritize reliability.

Alert on burn rate, meaning how quickly the budget is being used, instead of on single metric spikes. That reduces pages for blips that don't matter while still catching real incidents quickly.

A practical rollout plan

1. Start with auto-instrumentation for HTTP, gRPC, and database clients in every service.
2. Deploy an OpenTelemetry Collector as the single pipeline for batching, sampling, and export.
3. Standardize service names, environment tags, and log field names.
4. Build one RED dashboard per service from a shared template.
5. Define SLOs for two or three critical user journeys and alert on burn rate.
6. Put trace IDs in error responses so support tickets can link straight to a trace.

Summary

Metrics tell you something is wrong, traces show where, and logs explain why. Instrument once with OpenTelemetry, connect the three signals with a shared trace ID, control cardinality and sampling, and tie alerts to SLOs that reflect real user experience. At that point, a 4-second checkout becomes a quick investigation.