Logging feels solved until your system spans a dozen services and a production incident lands at 2 am. Suddenly, logs written independently by each service don't connect. Timestamps are out of sync. Request context disappears at every service boundary. You have megabytes of output and nothing that tells you what actually happened. This is the logging problem in distributed systems, and it's different in kind from what a monolith demands.
Structured logging is the baseline, not the goal
Unstructured logs, plain text strings emitted by a printf call, are nearly useless at scale. They're hard to query, impossible to aggregate, and don't survive schema changes gracefully. Structured logging, emitting JSON or key-value pairs with consistent field names, is the minimum viable starting point for distributed systems.
The field names matter more than most teams realise. Pick a schema and enforce it across every service from day one. At minimum, every log event should carry: a timestamp in ISO 8601 UTC, a severity level, a service name, a version string, and a trace or correlation ID. Teams that let each squad define their own fields end up with four different spellings of "user_id" and no reliable way to join events.
Use a log library that enforces structure rather than relying on developer discipline. In Python, structlog makes structured output the default path. In Go, slog (added to the standard library in 1.21) does the same. The library choice isn't the point; consistent enforcement is.
Correlation IDs: the single most important practice
A correlation ID is a unique identifier generated at the entry point of a request and propagated through every downstream service call. Without one, you cannot reconstruct the path a request took through your system. With one, you can pull every log line touching that request from every service in a single query.
Generate the ID at your API gateway or load balancer, not inside the first application service. Pass it in a header (typically X-Correlation-ID or X-Request-ID) and extract it at each service boundary. Inject it into your logging context automatically so developers don't have to remember to include it on every log call. If they have to think about it, some log lines won't have it, and those will be the ones you need during an incident.
Teams adopting OpenTelemetry get this largely for free through trace context propagation. The W3C traceparent header carries trace ID, span ID, and flags across HTTP, gRPC, and message queue boundaries without custom code. It's worth understanding how observability differs from monitoring before committing to a particular propagation model, because the tooling choices upstream affect how much of this becomes automatic.
What to log, and what not to
Teams consistently over-log happy paths and under-log failure modes. A log statement that fires on every successful database read adds cost and noise. A log statement that fires when a circuit breaker opens is genuinely valuable. Default to logging at boundaries: service entry and exit, external calls, state transitions, and errors.
Three things that should always appear in a log event tied to an error:
- The error type and message, without wrapping it in a generic catch-all string that discards the original
- Enough request context to reproduce or narrow down the triggering input
- The correlation ID, so the event can be joined to the surrounding trace
One thing that should never appear in a log: personally identifiable information. Australian organisations fall under the Privacy Act, and logging user emails, tax file numbers, or health identifiers into a system with broad access is both a compliance risk and a breach waiting to happen. Build PII scrubbing into your logging middleware, not as an afterthought during a security review.
Log levels are a contract, not decoration
Most teams use log levels inconsistently. DEBUG means "I put this here while debugging and forgot to remove it." ERROR fires for anything that looked alarming to the developer at the time. This breaks downstream alerting. When ERROR fires for non-critical validation failures, on-call engineers learn to ignore ERROR and start reading every line manually, which defeats the point.
Treat log levels as a contract with your operations tooling. ERROR should mean: something failed that a human needs to know about, now. WARN should mean: something degraded but the system is still functioning, and a pattern of these warrants investigation. INFO should mean: a meaningful business or system event occurred. DEBUG should be disabled in production by default and enabled only under sampling or on-demand.
The discipline required here is the same as defensive coding habits more broadly: consistency enforced by convention, not willpower.
Sampling: how to log at scale without going broke
At sufficient traffic volume, logging every request at INFO is expensive. A system handling 10,000 requests per second produces 864 million log events per day. Storage, ingestion, and query costs grow linearly with volume, and most of those events are identical happy paths you'll never read.
Head-based sampling decides at the trace entry point whether to log the full request. Tail-based sampling collects all spans and logs only traces that hit a threshold (high latency, an error, a slow dependency call). Tail-based sampling is more useful because it keeps the interesting cases and discards the boring ones, but it requires a collector that can buffer and evaluate spans before writing them, such as the OpenTelemetry Collector with a tail sampling processor.
Always log 100% of errors regardless of your sampling rate. Sampling is a cost-reduction tool for normal traffic. An error that gets sampled away is an incident you can't investigate.
Centralised log aggregation and what teams get wrong
Distributed systems need centralised log storage. Logs distributed across individual service pods are not queryable at incident time. The standard options for Australian teams are managed services: AWS CloudWatch Logs, Azure Monitor Logs, Google Cloud Logging, or a self-managed Elasticsearch or OpenSearch cluster.
The common mistake is treating log aggregation as solved once the pipeline is plumbed. It isn't. Log schemas drift. A new service emits a slightly different format for its timestamp. Retention policies get set once and never revisited. Ingestion costs spike when a noisy service starts logging at DEBUG in production.
Set log retention based on your incident investigation window and compliance obligations. For most Australian organisations under the Privacy Act's notifiable data breach scheme, 90 days is a practical minimum for security-relevant logs. Some industries require 7 years. Those are very different storage cost profiles, and conflating them is a budget problem waiting to happen.
Testing your logging
Most teams never test their logging. They ship code, check that something appears in the console, and move on. In production, they discover that the correlation ID wasn't propagated to the async worker queue, or that the error logger swallowed the stack trace.
Include log output assertions in your integration tests. Verify that a failed payment event emits an ERROR with the correct fields. Verify that a successful request emits an INFO with the correlation ID matching the injected header. This is unglamorous but catches the class of silent logging failure that only surfaces during an incident, when the logs you needed simply aren't there.
For teams exploring more rigorous code quality practices, mutation testing can expose whether your test suite actually validates log output or merely tolerates its absence. The underlying principle is the same: coverage that doesn't assert behaviour isn't coverage.