| license | Apache-2.0 |
| name | log-aggregation-architect |
| description | Centralized log pipeline architect with structured logging, Fluentd/Vector, and retention policies. Activate on: log aggregation, structured logging, Fluentd, Vector, Loki, ELK stack, log pipeline, log retention, centralized logging. NOT for: metrics and dashboards (use monitoring-stack-deployer), distributed tracing (use logging-observability), alerting rules (use site-reliability-engineer). |
| allowed-tools | Read,Write,Edit,Bash(docker:*,kubectl:*,terraform:*,npm:*,npx:*) |
| category | DevOps & Infrastructure |
| tags | ["logging","observability","fluentd","vector"] |
| pairs-with | [{"skill":"logging-observability","reason":"Broader observability patterns that log aggregation implements"},{"skill":"monitoring-stack-deployer","reason":"Metrics and logs often share infrastructure and correlation"}] |
Log Aggregation Architect
Expert in designing centralized log pipelines with structured logging, efficient collection, and cost-effective retention.
Activation Triggers
Activate on: "log aggregation", "structured logging", "Fluentd config", "Vector pipeline", "Loki setup", "ELK stack", "log pipeline", "log retention policy", "centralized logging", "log shipping"
NOT for: Metrics/dashboards → monitoring-stack-deployer | Distributed tracing → logging-observability | Alerting → site-reliability-engineer
Quick Start
- Standardize log format — JSON structured logs with consistent fields across all services
- Deploy collection agents — Vector or Fluentd as DaemonSet on every node
- Choose storage backend — Grafana Loki (cost-effective), Elasticsearch (full-text search), or ClickHouse (analytics)
- Define retention tiers — hot (7d searchable), warm (30d compressed), cold (1y archived)
- Build correlation — trace ID propagation so logs link to traces and metrics
Core Capabilities
| Domain | Technologies |
|---|
| Collection | Vector 0.43, Fluentd 1.17, Fluent Bit 3.2, OTEL Collector |
| Storage | Grafana Loki 3.x, Elasticsearch 8.x, ClickHouse, S3 archive |
| Structured Logging | JSON, logfmt, OpenTelemetry Logs, pino, winston, slog (Go) |
| Pipeline | Transform, filter, route, sample, deduplicate, redact PII |
| Visualization | Grafana (Loki), Kibana (Elastic), Grafana Explore |
Architecture Patterns
Vector Pipeline (Recommended 2026)
[sources.kubernetes]
type = "kubernetes_logs"
auto_partial_merge = true
[transforms.structured]
type = "remap"
inputs = []
=
=
= []
=
= [, , ]
=
= []
=
=
=
=
