| name | observability-queries |
| description | Canonical LogsQL, PromQL, and Jaeger-compatible trace queries against the harness Victoria stack. Use when investigating a bug via logs, metrics, traces, building an evidence bundle, or wiring a new alert. Includes copy-pasteable curl examples and gotchas: severity, not level; case-sensitive enum; mandatory `| fields` projection. |
Observability queries: the 6 shapes you'll actually use
Discover stack identity and ports
Run from the repository root after just bootstrap:
just stack inspect
eval "$(just stack ports)"
ISO=$(just stack inspect | awk -F: '/Iso key:/ {gsub(/^ +/, "", $2); print $2}')
export OTEL_EXPORTER_OTLP_ENDPOINT="http://localhost:${OTEL_OTLP_PORT}"
just stack inspect is the human-readable status entrypoint. just stack ports prints shell-safe KEY=value lines for the current worktree, including:
VL_PORT=...
VM_PORT=...
VT_PORT=...
OTEL_OTLP_PORT=...
If just is not available in an automation context, use the slot binary directly:
eval "$(harness/stack/bin/stack ports)"
ISO=$(harness/stack/bin/stack inspect | awk -F: '/Iso key:/ {gsub(/^ +/, "", $2); print $2}')
Do not use pnpm harness inspect in this template. That lab command is not a root package script here. Host-run apps should set OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:${OTEL_OTLP_PORT} before emitting telemetry.
Universal gotchas
severity, not level. Filtering level:error silently returns empty.severity is case-sensitive. Pino emits INFO and ERROR uppercase. Match severity:ERROR or use the union severity:(ERROR OR error) to handle mixed sources.- LogsQL has no
|~ regex-pipe operator. Use field:/regex/ or bare "word" for text match.
- Always project with
| fields .... Without it each log returns much larger rows. The projection is mandatory for any query touching more than 10 rows.
- OTEL to Prom attribute conversion: dots become underscores.
service.name in logs becomes service_name in PromQL labels. Metric names keep dots in VictoriaMetrics, for example http.server.duration_bucket, and must be selected as {__name__="http.server.duration_bucket"}.
| fields drops _stream and _stream_id. If you need full context for one line, re-query that _time without projection.
Q1: LogsQL error scan with | fields projection
Returns: recent error log lines, projected to 4 fields.
curl -sG "http://localhost:${VL_PORT}/select/logsql/query" --data-urlencode \
"query={harness.iso_key=\"$ISO\"} severity:ERROR | fields _time, service.name, severity, _msg | limit 20"
Sample response, one NDJSON result per line:
{"_time":"2026-05-14T09:09:10.084033844Z","service.name":"api-cpp","severity":"ERROR","_msg":"migration error: SQL execution timeout"}
Q2: LogsQL structured-field filter by service.name
Returns: logs scoped to one service. Use for "what was API doing at T?"
curl -sG "http://localhost:${VL_PORT}/select/logsql/query" --data-urlencode \
"query={harness.iso_key=\"$ISO\",service.name=\"api\"} | fields _time, severity, _msg | limit 20"
Sample:
{"_time":"2026-05-14T09:09:12.912Z","severity":"info","_msg":"Server listening at http://0.0.0.0:3000"}
The stream selector {service.name="api"} is cheaper than the post-filter _stream:{service.name="api"} because it filters at the index.
Q3: LogsQL by traceID
Returns: every log line emitted under one trace_id. This is the critical step in trace to logs correlation.
TRACE_ID=d17a6f60131d4c23162495a8e4ae5bc3
curl -sG "http://localhost:${VL_PORT}/select/logsql/query" --data-urlencode \
"query={harness.iso_key=\"$ISO\"} trace_id:$TRACE_ID | fields _time, severity, _msg, req.url, res.statusCode | limit 50"
Sample:
{"_time":"2026-05-14T16:48:00.408Z","trace_id":"d17a6f60131d4c23162495a8e4ae5bc3","_msg":"incoming request"}
{"_time":"2026-05-14T16:48:00.409Z","trace_id":"d17a6f60131d4c23162495a8e4ae5bc3","_msg":"request completed"}
Get the trace_id from a failed Playwright response: response.request().headers()['traceparent'], then take the second hex segment.
Q4: PromQL error rate
Returns: instant rate of requests grouped by HTTP status over the last 5 minutes.
curl -sG "http://localhost:${VM_PORT}/api/v1/query" --data-urlencode \
'query=sum by (http.status_code) (rate({__name__="http.server.duration_count"}[5m]))'
Sample:
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"http.status_code":"200"},"value":[1778778648,"0.199"]}]}}
The metric name http.server.duration_count keeps OTEL dots, so select it through {__name__="..."}. Raw http.server.duration_count is a PromQL syntax error.
Q5: PromQL latency p95
Returns: 95th-percentile HTTP server latency over the last 5 minutes.
curl -sG "http://localhost:${VM_PORT}/api/v1/query" --data-urlencode \
'query=histogram_quantile(0.95, sum by (le) (rate({__name__="http.server.duration_bucket"}[5m])))'
Sample:
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1778778648,"4.749"]}]}}
Add , service.name to the sum by (...) group to get per-service p95. Values are in milliseconds for the OTEL HTTP semantic convention default.
Q6: Jaeger-compatible recent traces by service name
Returns: the N most recent traces emitted by a given service.
curl -sG "http://localhost:${VT_PORT}/select/jaeger/api/traces" \
--data-urlencode "service=api" --data-urlencode "limit=10"
For one specific trace:
curl -s "http://localhost:${VT_PORT}/select/jaeger/api/traces/$TRACE_ID"
List services first if unsure:
curl -s "http://localhost:${VT_PORT}/select/jaeger/api/services"
VictoriaTraces exposes the Jaeger HTTP API, not raw TraceQL HTTP. This is by design. The compose template uses VictoriaTraces and the OTEL collector exports to /api/v2/spans.
Composed workflow: failed Playwright response to root-cause log
- Playwright catches a
requestfailed event or 5xx response.
- Pull
traceparent from response.request().headers(); the trace_id is segment 2.
- Run Q3 to fetch the log lines for that trace_id.
- Run Q1 filtered to that service to pull surrounding ERROR lines for context.
- Run Q6 and open the trace in the Jaeger-compatible UI at
http://localhost:${VT_PORT}/select/jaeger/... to see span timing.
See .claude/skills/before-after-evidence/SKILL.md for the full trace-correlation recipe.
Verification
These query shapes were ported from the lab observability skill and adapted to this template's stack-manager entrypoints. For an executable end-to-end smoke against this template, run:
just observability-smoke
