_time:2024-03-15T10:00:00+02:00 — exact RFC3339 with timezone.
Grafana plugin and web UI inject the dashboard time range. Don't add _time: in dashboard queries — disables time
picker. Do add it for ad-hoc API queries.
Stream Filter — The Performance Lever
Logs belong to streams — Prometheus-style label sets identifying the source. Use {...} braces with PromQL selector
syntax:
Byte length of field value. For suspiciously short or long lines.
Logical Composition
error AND _time:5m AND {app="nginx"}
error _time:5m {app="nginx"} # AND can be omitted
error AND NOT buggy_app
error -buggy_app # - is shorthand for NOT
(error OR warn) AND _time:1h
error -(buggy_app OR foobar)
Precedence: NOT > AND > OR. Parenthesize when in doubt — error -buggy_app OR foobar parses as
(error AND NOT buggy_app) OR foobar.
The * Wildcard for Fields
Multi-field search by name prefix:
*:error # any field containing 'error' word
kubernetes.*:nginx # any field starting with 'kubernetes.' containing 'nginx'
Slows queries — scans more field data. Prefer explicit field names.
Pipe Composition
Pipes chain with |.
fields — Select Output Fields
error _time:5m | fields _time, _stream, _msg
Reduces to named subset — faster transit, easier reading.
sort — Order Results
error _time:5m | sort by (_time)
error _time:5m | sort by (_time desc)
Unordered by default. Expensive — only sort small sets (< few million).
limit / first / last
error _time:5m | sort by (_time) desc | limit 10
error _time:5m | first 10 by (_time desc)
error _time:5m | last 10 by (_time)
limit truncates after sort. first/last combine sort+limit. Aliases: head = limit, skip = offset.
stats — Aggregate
_time:5m | stats count() as total_errors
_time:5m | stats by (_stream) count() as errors_per_stream
_time:5m | stats by (_time:1m) count() as per_minute_errors
stats and as keywords are optional: _time:5m | count() total_errors works.
Most powerful pipe. Grouping by fields, time buckets, field buckets, IPv4 buckets, per-row filters. Full mechanics:
[${CLAUDE_SKILL_DIR}/references/stats.md].
{app="my-app"} | unpack_json from _msg
{app="my-app"} | unpack_logfmt from _msg
{app="my-app"} | unpack_json from _msg | stats by (level) count()
Parses field as JSON/logfmt, flattens into top-level fields. Downstream pipes filter and group by extracted fields.
extract / extract_regexp — Pattern-Based Field Extraction
_msg:"GET" | extract "GET <path> HTTP" from _msg
_msg:"client" | extract_regexp "client (?P<client_ip>[0-9.]+)" from _msg
extract uses <field_name> placeholders. extract_regexp uses RE2 named groups.
math / eval — Numeric Calculations
| math duration_ms / 1000 as duration_sec
| eval (errors / total) * 100 as error_percent
Arithmetic, bitwise, defaults, max/min, abs/ceil/floor/round/exp/ln on numeric field values.
top — N Largest Groups
_time:5m | top 10 (_stream)
_time:5m | top 10 (host, path)
Shortcut for stats by (group) count() | sort desc | limit N.
uniq — Distinct Rows
_time:5m | uniq by (host, path)
Like SQL SELECT DISTINCT. One row per unique field combination.
Common Query Recipes
# Recent error logs from one service
_time:5m {app="api"} error | sort by (_time) desc | limit 100
# Error count per service over last hour
_time:1h error | stats by (_stream) count() as errors | sort by (errors) desc
# Per-minute error rate
_time:1h error | stats by (_time:1m) count() as errors_per_min
# Top 10 noisiest streams
_time:5m | stats by (_stream) count() as logs | sort by (logs) desc | limit 10
# Equivalent:
_time:5m | top 10 (_stream)
# Latency percentiles per endpoint
_time:1h {app="api"} | unpack_json from _msg
| stats by (path) quantile(0.5, duration_ms) p50, quantile(0.95, duration_ms) p95
# Errors from JSON logs, parsed and filtered
{app="api"} _time:5m | unpack_json from _msg
| filter level:error
| stats by (component) count()
Stats Functions — Quick Reference
Most common; full catalog: [${CLAUDE_SKILL_DIR}/references/stats.md].
count() — row count.
count(field) — rows with non-empty field.
count_empty(field) — rows with empty/missing field.
count_uniq(field, ...) — distinct combinations. Memory scales with cardinality.
count_uniq_hash(field, ...) — cheaper approximate distinct via hashing.
sum, avg, min, max, median of field.
quantile(phi, field) — phi in [0..1].
uniq_values(field) — distinct values as JSON array.
values(field) — all values including duplicates.
field_min(target, sort) / field_max(target, sort) — target value for smallest/largest sort row.
row_min(sort) / row_max(sort) — full row at min/max.
row_any() — arbitrary row per group.
rate() — per-second rate of matching rows.
histogram(field) — histogram of numeric values.
stddev(field).
Grouping in Stats
By Fields
| stats by (host, path) count() as logs
by keyword optional.
By Time Buckets
| stats by (_time:1m) count() as per_minute
| stats by (_time:5m, host) count() as per_5m_per_host
Bucket = any duration. Named buckets: nanosecond, microsecond, millisecond, second, minute, hour, day,
week, month, year (month/year account for variable lengths).
Time Buckets with Timezone
| stats by (_time:1d offset 'America/New_York') count() as per_day_ny
Aligned to the named timezone, not UTC. Needed for daily/weekly stats in non-UTC offices.
By Field Buckets
| stats by (duration_ms:100) count() as histogram
Buckets numeric values into ranges. duration_ms:100 → [0,100), [100,200), etc.
With Additional Filters
| stats
count() as total,
count() if (level:error) as errors,
count() if (level:warn) as warns
Per-aggregate filtering. One pass beats N separate queries.
Performance Tips
Always include _time. Otherwise every block is candidate for scan.
Use {...} stream filters when possible. Block-level pruning — orders of magnitude faster than field-value.
Exact over regex.log.level:="error" is much faster than log.level:~"^err".
Filter before aggregating. Push filters left.
Avoid sorting > 10M rows. Combine with limit or use first N by (...).
count_uniq_hash over count_uniq for high cardinality when ±1% is acceptable.
fields early to drop unused columns.
Stream filter at top level. Multiple stream filters work but are less optimized.
