observability

name	observability
description	Observability — forge doctor, Grafana dashboards, querying logs/traces/metrics.

Observability

Every Forge project ships with a full observability stack running locally via Docker Compose (Grafana LGTM: Grafana, Prometheus, Tempo, Loki, Pyroscope). No external services needed.

forge doctor

Verify the entire observability pipeline is working:

forge doctor                    # Check everything
forge doctor --signal traces    # Check only traces
forge doctor --signal metrics   # Check only Prometheus
forge doctor --signal logs      # Check only Loki
forge doctor --signal profiles  # Check only profiling
forge doctor --json             # Machine-readable output
forge doctor --verbose          # Show evidence for passing checks

Doctor checks: Docker containers running, app health endpoint, pprof endpoint, Prometheus targets up, Tempo traces ingested, Loki log streams present, Pyroscope profiles available.

Run forge doctor after forge up --env=dev to verify the pipeline is healthy before investigating issues.

forge doctor verifies the telemetry pipeline, NOT app-flow correctness. It (like forge smoke) is green when containers, endpoints, and signal ingestion are healthy — it can be green while the actual app flow is broken (e.g. a cross-cluster dial failing). To prove an app-flow invariant holds, use a declarative, exit-coded app-health assertion (model: a project doctor:<flow> task) plus a full forge test e2e. Doctor tells you observability works; it does not certify the app does.

Accessing Grafana

Grafana's port is dynamically assigned. Find it with:

docker compose ps    # Look for the lgtm container's port mapping
forge doctor         # Also shows the Grafana URL

Two dashboards are auto-provisioned:

• Logs Dashboard — Log volume by level, filterable by service/container
• Traces Dashboard — Trace search, latency distribution, service map

Querying Logs (Loki)

In Grafana → Explore → Loki, use LogQL:

{container=~".*app.*"} | json | level="error"
{container=~".*app.*"} | json | procedure="/services.users.v1.UsersService/Create"
{container=~".*app.*"} | json | trace_id="abc123"

Logs are structured JSON with consistent attribute keys (procedure, request_id, trace_id, duration_ms, user_id, status, code). Emit the same attribute keys from your own log sites so dashboards stay queryable.

Querying Traces (Tempo)

In Grafana → Explore → Tempo, search by:

• Service name (matches your project name)
• Trace ID (from log lines — click a trace_id value to jump to the trace)
• Duration range
• Status code

Trace IDs are automatically injected into every log line, connecting logs to traces.

Querying Metrics (Prometheus)

In Grafana → Explore → Prometheus, use PromQL:

go_goroutines                              # goroutine count per service
up                                         # scrape targets status
rate(http_request_duration_seconds_count[5m])  # request rate

Rules

• Run forge doctor after forge up --env=dev to verify observability before investigating issues.
• Grafana port is dynamically assigned — use docker compose ps or forge doctor to find it.
• The alloy-config.alloy and dashboard files are regenerated by forge generate — do not hand-edit.
• Use the logevents.go helpers for structured log events — do not create ad-hoc attribute keys.
• Trace IDs propagate automatically via OpenTelemetry context — no manual instrumentation needed.