Use when designing log schemas, choosing JSON vs text logs, setting up correlation IDs across services, redacting PII, controlling log volume costs, or wiring logs into OpenTelemetry. Triggers: "logs are unsearchable", trace_id missing from logs, PII leak in production logs, log volume bill spike, log levels misused, structured fields vs string interpolation, parent_span_id propagation, sampled vs always-log decisions, log routing (vendor + cold storage). NOT for log aggregation tooling specifically (vendor skills), full APM, or print-debugging local development.
Use when designing log schemas, choosing JSON vs text logs, setting up correlation IDs across services, redacting PII, controlling log volume costs, or wiring logs into OpenTelemetry. Triggers: "logs are unsearchable", trace_id missing from logs, PII leak in production logs, log volume bill spike, log levels misused, structured fields vs string interpolation, parent_span_id propagation, sampled vs always-log decisions, log routing (vendor + cold storage). NOT for log aggregation tooling specifically (vendor skills), full APM, or print-debugging local development.
allowed-tools
Read,Grep,Glob,Edit,Write,Bash
metadata
{"category":"DevOps & Infrastructure","tags":["logging","observability","structured-logs","json","correlation","pii"],"provenance":{"kind":"first-party","owners":["port-daddy"]},"pairs-with":[{"skill":"logging-observability","reason":"The broader observability wiring (collectors, pipelines, alerting) that this skill's log schema feeds into."},{"skill":"log-aggregation-architect","reason":"Owns the aggregation/routing layer (Vector, Fluent Bit, hot vendor + cold storage fan-out) sketched in this skill's Routing section."},{"skill":"observability-apm-expert","reason":"Trace/span correlation designed here joins logs to APM traces; that skill owns the tracing side of the join."}],"io-contract":{"kind":"deliverable","consumes":["[Truncated]","[Truncated]"],"produces":["[Truncated]","[Truncated]"]}}
Structured Logging Design
Logs are searched in production, not read top-to-bottom. Structured (JSON) logs with consistent field names are queryable; unstructured strings are grep bait. The whole craft is choosing the schema, the levels, and what to redact — so you can answer "what happened to user X at 03:14" without guessing.
Decision diagram
flowchart TD
A[Need to add a log line] --> B{Is data in the message string or structured fields?}
B -->|String interpolation| F1[FIX: move data to fields, message stays static]
B -->|Structured fields| C{Field name in PII redact list?}
C -->|Yes, raw| F2[FIX: add to redactor before logger writes]
C -->|No| D{Has trace_id from request context?}
D -->|No| F3[FIX: bind trace_id via middleware/contextvars]
D -->|Yes| E{Field high-cardinality e.g. user_id in path?}
E -->|Yes, indexed| F4[FIX: normalize to template, raw in non-indexed payload]
E -->|Normalized| G{Long body field truncated?}
G -->|No| F5[FIX: cap at 1KB unless level=debug]
G -->|Yes| H[Log it]
H --> I{Service+version+region in base context?}
I -->|No| F6[FIX: enricher in logger init, not per-call]
I -->|Yes| J[Done]
This keeps debug noise local to the small sample of traces being kept anyway.
Cost-conscious patterns
Cardinality awareness. A field like request_path with /users/u_42/posts creates as many search indices as users. Normalize to /users/:id/posts for indexing; keep raw path in a non-indexed field.
Long fields. Truncate request_body, response_body to 1KB (or omit unless level=debug). Otherwise one big payload doubles the bill.
Repeated context.service, version, region come from environment, not the log call. Set them as enrichers once.
Set up the agent or sidecar (Vector, Fluent Bit) to fan out. Hot vendor for searchable last 7 days; cold storage for compliance and "where was this user 6 months ago."
Anti-patterns
String interpolation instead of fields
Symptom: Logs are searchable by full-text but not faceted; redaction misses.
Diagnosis:log.info(f"user {email} did X") puts data in the message body.
Fix:log.info('user did X', user_email=email, action='X'). Fields go in fields.
Inconsistent field names
Symptom: Some logs use userId, some user_id, dashboards split.
Diagnosis: No schema standard.
Fix: Pick a convention (snake_case is common for JSON), document, lint.
High-cardinality fields indexed
Symptom: Logging vendor bill is mostly index cost.
Diagnosis:request_path with raw IDs, error_message with timestamps embedded.
Fix: Normalize. Move high-cardinality to non-indexed payload fields.
Errors logged twice
Symptom: Same error appears N times in logs; alerts fire N times.
Diagnosis: Each layer (handler, middleware, top-level) logs the same exception.
Fix: Log at one well-defined layer (top-level error handler). Lower layers re-throw with context.
PII in logs
Symptom: Compliance audit finds emails in production logs.
Diagnosis: No redactor; engineers add fields without thinking.
Fix: Logger-level redaction by field name. Block list updated when new PII fields are added. CI grep for known PII names.
Synchronous I/O in the log path
Symptom: A slow log destination slows down the request.
Diagnosis: Logger writes to stdout that's piped to a synchronous shipper.
Fix: Async logger with a buffer. Drop on overflow rather than block. Pino, slog, structlog all default to this.
Worked example: the logging bill spike
Scenario. Datadog Logs bill went from $2k/mo to $11k/mo over 4 weeks. Volume looks the same. Finance is asking questions.
Novice would: Drop the retention period, sample debug logs at 10%, ship a one-line config change. Bill drops 30% but search becomes useless: queries that used to find "what happened to user X" now miss because the log was sampled out.
Expert catches:
The bill is index cost, not volume cost. Open Datadog usage page → break down by indexed-bytes vs ingested-bytes. If indexed is 10x ingested, the cost is from facet cardinality.
Find the high-cardinality field. Run top facets by unique values. Usually it's request_path with raw IDs, or error.message with embedded timestamps/UUIDs.
Normalize, don't sample. Add a request_path_template field (/users/:id/posts) that gets indexed, keep raw request_path in payload (un-indexed, still searchable via grep at 1/10 the cost). Same volume, ~70% cost cut.
Cold storage for compliance. The 2-year retention for "who did what 14 months ago" goes to S3 + Athena, not the hot vendor. 5x cheaper, slower query, fine for the use case.
Timeline. Novice's sampling fix passes finance review but breaks the next forensics request ("we can't tell when the breach started"). Expert's normalization + tiered storage fix passes finance review AND keeps every log searchable, just at different latencies. Bill stabilizes at $3k/mo.
Quality gates
Test: schema-conformance test asserts every log line in a sample has ts, level, service, message, trace_id (when request-scoped).
Test: PII fuzz — generate logs with fake emails/SSNs in known fields, assert redactor replaces them. Run in CI.
All production log lines are JSON. CI lint fails on console.log / print() outside dev paths.
Field-name convention picked (snake_case or camelCase) and enforced via a lint rule (eslint-plugin-no-mixed-keys or similar).
Top-level error handler is the single error log site. Lower layers re-throw with context. Verified by grep for log.error count per service (should be ≤ 3 per layer).
Long fields capped at 1KB. Verified by checking p99 log line size ≤ 4KB in vendor metrics.
Log volume budget per service tracked in grafana-dashboard-builder panel; alert at 2x baseline for 30 minutes.
Cold storage retention is separate from hot search retention; documented in runbook.
trace_id matches the OTel exporter's trace ID (W3C traceparent) — verified by joining a log line and a span in the vendor UI on a recent request.
Async logger configured (Pino/slog/structlog defaults); verified handler latency ≤ same with logger disabled.
Deterministic Audit
Before committing to a logging design (or reviewing another agent's), write it as a
JSON plan matching schemas/logging-plan.schema.json and run the deterministic auditor:
auditStructuredLogging(plan) (in scripts/structured_logging_audit.mjs) turns this
skill's anti-patterns and Quality Gates into machine-checkable rules over structured
fields — no keyword matching: text logs in production, data interpolated into message
strings, no logger-level PII redactor, missing trace_id binding, mixed field-name
conventions, high-cardinality fields left indexed, errors logged at every layer, a
synchronous log path, and uncapped long fields. It returns
{ pass, score, findings, recommendations }. examples/sample-input.json is a
JSON-logs plan with redaction, correlation, and tiered routing (pass: true). Version
history lives in CHANGELOG.md.
NOT for
Vendor-specific dashboards — different layer. → grafana-dashboard-builder for Grafana/Loki visualization.
Full APM (traces + metrics + logs as one system) — logs are part of observability, not the whole. → opentelemetry-instrumentation for the instrumentation side.
Print-debugging in dev — console.log is fine; this skill is for production. No dedicated skill needed.
Audit/compliance logs — separate concerns (immutable storage, signing). No dedicated skill.
Log alerting rules — once logs are structured, alerting on them. → grafana-dashboard-builder (alerting section).
