Logs

Metrics told you that something is wrong. Logs tell you what exactly happened — the detailed, discrete record of events. This post covers what logs are, the pivotal shift from unstructured text to structured logging, log levels, what to log (and what never to), and how logs work at scale. Logs are the most familiar pillar and the one most teams do badly, because habits from single-machine printf debugging don’t survive contact with distributed systems.

What logs are

A log is a timestamped record of a discrete event — “user 42 logged in,” “payment failed: card declined,” “request completed in 230ms.” Where metrics aggregate (millions of events into a few numbers), logs are individual: each entry is one event, with as much detail as you choose to include. That’s their strength — they carry the specific what happened here that aggregated metrics discard — and their weakness — they’re voluminous and expensive at scale.

Logs answer the questions metrics can’t: not “the error rate is up” but “this specific error, with this stack trace, for this user, doing this operation.” When you’ve detected a problem with metrics, logs are where you often find the actual cause. The catch is that a pile of unstructured text logs is nearly impossible to search and analyze at scale — which is what structured logging fixes.

The big shift: structured logging

The single most important practice in this post: log structured data, not human-readable strings. Traditional logging emits text meant for a person to read:

Unstructured:
  2026-08-16 10:32:01 ERROR Payment failed for user 42 on order 1001: card declined

Structured (JSON):
  {"ts":"2026-08-16T10:32:01Z","level":"error","event":"payment_failed",
   "user_id":42,"order_id":1001,"reason":"card_declined","service":"checkout"}

Both carry the same information, but they’re worlds apart operationally:

• The text log is for grep; the structured log is for queries. With structured logs you can ask “all payment_failed events with reason=card_declined in the last hour, grouped by service” — a precise query. With text logs you’re writing fragile regexes and hoping the format is consistent.
• Structured logs are machine-parseable, so log aggregation systems can index, filter, and aggregate them reliably. Text logs require brittle parsing that breaks when the message format changes.
• Fields enable correlation — a structured log with a trace_id field links to the trace it belongs to (next post), and consistent field names let you join across services.

At the scale of a distributed system, you don’t read logs line by line — you query them, and only structured logs support that. Adopting structured logging (JSON or a structured format, via a logging library) is the highest-leverage logging change most teams can make. Text logs are a single-machine habit; structured logs are the distributed-systems requirement.

Log levels: signal over noise

Logs use levels to indicate severity and control volume, and using them consistently is what keeps logs useful rather than a wall of noise:

• ERROR — something failed that needs attention; a real problem.
• WARN — something unexpected but handled; worth noticing, not an emergency.
• INFO — significant normal events (service started, request completed); the default useful narrative.
• DEBUG — detailed diagnostic information, usually off in production, enabled when investigating.

The disciplines that matter: be consistent (an ERROR should mean a real error, not a routine event, or alerts on ERROR become meaningless), and control volume by level (run production at INFO, enable DEBUG selectively when investigating). Level misuse — logging routine events at ERROR, or everything at INFO — drowns the signal you need in noise, and noisy logs are ignored logs.

What to log, and what never to log

What you put in logs is both an observability and a security decision:

• Log enough context to be useful — include the identifiers that let you correlate and investigate: trace_id/request_id (to link to traces and group a request’s logs), user/tenant ID, the operation, relevant parameters, and for errors the full error and stack. A log without context (“something failed”) is nearly useless.
• Never log secrets or sensitive data — this is critical and constantly violated. Passwords, tokens, API keys, full credit-card numbers, and personal/sensitive data must never go in logs. Logs are widely accessible, retained, and shipped to third-party aggregators, so a secret in a log is a secret leaked. Redact or omit sensitive fields deliberately. (Especially relevant for the privacy-sensitive apps this blog discusses — logging user content can undo an on-device/local-first privacy promise.)
• Log events, not narration — log meaningful state changes and decisions, not a play-by-play of every line executed; the latter is noise and cost.

The two rules together: rich operational context (IDs, operations, errors) so logs are useful, and zero sensitive content so they’re safe.

Logs at scale: aggregation and cost

In a distributed system, logs from many services and instances must be centralized — shipped to a log aggregation system where they’re indexed and queryable in one place. You cannot SSH into dozens of ephemeral containers to read logs; they must flow to a central store. This is standard, but it surfaces two realities:

• Correlation requires consistent fields — centralized logs are only powerful if you can join them, which needs consistent structured fields (especially a shared trace_id) across all services. This is why structured logging and trace-context (next post) go together.
• Logs are expensive — they’re voluminous, and log storage/ingestion is often a major observability cost. This forces discipline: log at appropriate levels, sample very high-volume logs, set retention policies, and don’t log noise. Unlike cheap aggregated metrics, logs cost roughly in proportion to volume, so “log everything forever” is financially real. Log deliberately: enough to investigate, not so much that cost and noise overwhelm the value.

Logs are the detailed what-happened layer — indispensable for investigation, powerful when structured, dangerous when they carry secrets, and costly at volume. But logs alone don’t show how an event connects to the rest of a request’s journey across services. That connective view is the third pillar: traces.

Key takeaways

• A log is a timestamped record of a discrete event carrying detailed context — the what exactly happened that aggregated metrics discard — indispensable for finding the actual cause after metrics detect a problem.
• The pivotal shift is structured logging (machine-parseable JSON/fields) over human-readable text: structured logs are queryable (“all payment_failed with reason=card_declined, grouped by service”) and correlatable, while text logs need fragile regex parsing — this is the highest-leverage logging upgrade.
• Use log levels consistently (ERROR=real failure, WARN=handled-unexpected, INFO=significant normal events, DEBUG=diagnostic, off in prod) to keep signal above noise; level misuse makes logs and their alerts meaningless.
• Log rich operational context (trace_id/request_id, user/tenant, operation, full errors) so logs are useful — but NEVER log secrets or sensitive data (passwords, tokens, cards, PII), because logs are accessible, retained, and shipped externally.
• At scale, centralize logs to a queryable aggregation system with consistent fields (a shared trace_id) for correlation, and manage cost deliberately (levels, sampling, retention) since log volume drives a major share of observability spend.

Further reading

• Metrics (previous post)
• OpenTelemetry — logs
• Web Identity: securing identity — why secrets must not leak into logs