PromQL query writing: selectors, operators, vector matching, functions, aggregations, histograms, subqueries, recording rules, alerting rules, and query optimization. Invoke whenever task involves any interaction with PromQL — writing, debugging, optimizing, reviewing, or explaining Prometheus queries for dashboards, alerts, recording rules, or ad-hoc exploration.
PromQL
PromQL is a nested functional language for selecting and aggregating Prometheus time series. Every query is an
expression tree: selectors at the leaves, functions and operators at the nodes, a single root value (scalar, instant
vector, range vector, or string) at the top. Type discipline makes queries correct; cardinality awareness makes them
fast.
Authoritative reference for writing PromQL queries — dashboards, alerts, recording rules, exploration. The companion
backend/prometheus skill covers instrumentation (metric types, naming, labels, exporters). When both apply, use
backend/prometheus for emitting metrics and this skill for querying them.
References
Selectors and data types — [${CLAUDE_SKILL_DIR}/references/selectors-and-types.md] Expression types, instant and
range vector selectors, label matchers (=, !=, =~, !~), time durations, offset and @ modifiers, staleness,
instant vs range queries, string and float literals
— [] Full function catalog: counter family, gauge
functions, classic and native histograms, , existence checks, label manipulation, math, time, type
conversion, sorting, trigonometric
Functions
${CLAUDE_SKILL_DIR}/references/functions.md
*_over_time
Native histograms — [${CLAUDE_SKILL_DIR}/references/native-histograms.md] Native vs classic detection,
histogram_quantile across both forms, native-only functions (histogram_avg, histogram_count, histogram_sum,
histogram_fraction, histogram_stddev, histogram_stdvar), aggregation without le, NHCB semantics, trim
operators, migration patterns, function compatibility matrix, gotchas, annotations
Subqueries — [${CLAUDE_SKILL_DIR}/references/subqueries.md] Syntax, resolution and alignment, nested subqueries,
when to use subqueries vs recording rules, pitfalls
Recording and alerting rules — [${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md] Rule file structure,
level:metric:operations naming, ratio aggregation patterns, alert syntax, for/keep_firing_for, templating,
alerting best practices, anti-patterns
Read the relevant reference before proceeding.
Data Types
Every expression has one type. Functions and operators require specific input types — type mismatch is the most common
cause of "query parses but returns nothing".
instant vector — set of series, one sample per series at one timestamp. Default result of metric selectors,
aggregations, most functions.
range vector — set of series, multiple samples per series across a time window. Produced only by range vector
selectors (metric[5m]) and subqueries. Cannot be graphed directly — must pass through a function returning an
instant vector.
scalar — single numeric value, no labels. Used as numeric arguments and in arithmetic.
string — function arguments only; cannot be a query result.
http_requests_total # all series with this metric name
http_requests_total{job="api"} # filtered by label
http_requests_total{job="api", method="GET"} # multiple matchers — AND
http_requests_total{status=~"5.."} # regex match (RE2, fully anchored)
http_requests_total{method!="OPTIONS"} # negative match
{__name__=~"http_.*"} # regex on metric name via __name__
Staleness: instant vector selectors return the most recent sample within the lookback window (default 5 minutes).
Series with no sample in the window disappear from the result.
Full selector reference, lookback delta tuning, instant-vs-range-query semantics:
[${CLAUDE_SKILL_DIR}/references/selectors-and-types.md].
Operators
Arithmetic and Comparison
+, -, *, /, %, ^ — IEEE 754 arithmetic. The metric name is dropped from any vector arithmetic result.
==, !=, >, <, >=, <= — by default filter: drop series where the comparison is false. Add bool after
the operator to return 0 or 1 instead.
http_requests_total > 100 # only series whose latest value exceeds 100
http_requests_total > bool 100 # all series, value is 1 if > 100 else 0
Logical/Set Operators
Operate on label-set membership, ignoring sample values:
vector1 and vector2 — intersection
vector1 or vector2 — union
vector1 unless vector2 — complement
up{job="prom"} or up{job="node"} # union
metric and on(device) (other_metric == 0) # constrained intersection
Vector Matching
The biggest source of "no results returned". Default rule: two elements match if they have exactly the same label
set (after the metric name drops). When operands have different label sets, match keywords are required:
on(labels) — match only on these labels; ignore all others
ignoring(labels) — match on all labels except these
Group modifiers for many-to-one (arithmetic, comparison, trigonometric only — not set operators):
group_left(extra_labels) — many on the left, one on the right; copy extra_labels from right to result
group_right(extra_labels) — symmetric
Canonical patterns:
# Error ratio: errors carry an extra `code` label; ignore it to match
errors:rate5m{code="500"} / ignoring(code) requests:rate5m
# Many-to-one: same denominator for every code
errors:rate5m / ignoring(code) group_left requests:rate5m
# Join info-style metric: bring `version` label into result
node_filesystem_avail_bytes * on(instance, job) group_left(version) node_exporter_build_info
Aggregation
Instant vector → instant vector with fewer elements.
sum, avg, min, max — reduce across dimensions
count — number of series; group — value 1 per group
topk(k, v) / bottomk(k, v) — k largest/smallest; preserves labels
stddev, stdvar — population statistics
quantile(φ, v) — quantile across series
count_values(label, v) — distribution of values
by vs without:
by (labels) keeps only the listed labels
without (labels) keeps everything except the listed labels
Prefer without for normal aggregations — preserves job, instance, and friends. Use by for a known output
dimension set (typically recording rules).
sum without (instance) (rate(http_requests_total[5m])) # collapses instance, keeps job/method/status
sum by (job) (rate(http_requests_total[5m])) # collapses everything except job
Operator Precedence (highest to lowest)
^
*, /, %, atan2
+, -
==, !=, <=, <, >=, >
and, unless
or
Left-associative except ^ (right-associative). Use parentheses when mixing comparison and logical operators.
Full operator catalog, fill modifiers, detailed vector matching examples:
[${CLAUDE_SKILL_DIR}/references/operators.md].
Key Functions by Category
Counters — rate, irate, increase
The three most important PromQL functions. All take a range vector, return an instant vector, handle counter resets
automatically.
rate(v[5m]) — per-second average rate over the range, extrapolated. Use for alerts and slow counters.
irate(v[1m]) — instant rate from the last two samples. Graphing volatile counters only; never alerts (flaps).
increase(v[1h]) — total increase over the range; syntactic sugar for rate(v) * range_seconds. Use for
human-readable totals.
Composition rule — rate() first, then aggregate:
sum(rate(http_requests_total[5m])) # CORRECT
rate(sum(http_requests_total)[5m]) # WRONG — counter resets disappear into the sum
Applies to all counter functions (rate, irate, increase), gauge functions (delta, deriv, predict_linear),
and *_over_time functions.
Window sizing: range must contain ≥2 samples. Rule of thumb: 4× the scrape interval, minimum 1m. With 15s scrapes
the standard default is [5m].
Gauges — delta, idelta, deriv, predict_linear
For gauges only. Never apply rate() to a gauge.
delta(cpu_temp_celsius[2h]) # absolute change over 2h
deriv(node_memory_MemAvailable_bytes[5m]) # smoothed per-second change
predict_linear(node_filesystem_avail_bytes[1h], 4*3600) < 0 # will fs fill in 4h?
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
# Aggregating across instances — `le` MUST be preserved
histogram_quantile(0.95, sum by (job, le) (rate(http_request_duration_seconds_bucket[5m])))
histogram_quantile(0.95, sum without (instance) (rate(http_request_duration_seconds_bucket[5m])))
# Average from sum/count
rate(http_request_duration_seconds_sum[5m]) / rate(http_request_duration_seconds_count[5m])
Native histograms are single composite series — quantile aggregation does not require le:
histogram_quantile(0.9, sum by (job) (rate(http_request_duration_seconds[5m])))
histogram_fraction(0, 0.2, rate(http_request_duration_seconds[5m])) # fraction ≤ 200ms
histogram_avg(rate(http_request_duration_seconds[5m]))
histogram_count(rate(http_request_duration_seconds[5m])) # rate of observations
histogram_sum(rate(http_request_duration_seconds[5m]))
If native histograms are available, prefer them — fewer series, easier aggregation, lower error.
Full native histogram semantics — detection, function compatibility matrix, NHCB behavior, trim operators, migration,
gotchas: [${CLAUDE_SKILL_DIR}/references/native-histograms.md].
Aggregation Over Time — *_over_time
Range vector → one summary value per series.
avg_over_time(range-vector) — average across the range
min_over_time / max_over_time — extremes
sum_over_time / count_over_time — totals
last_over_time / present_over_time — most recent / existence
quantile_over_time(φ, range-vector) — quantile across time
stddev_over_time / stdvar_over_time
avg_over_time(node_memory_MemAvailable_bytes[1h])
max_over_time(rate(http_requests_total[5m])[1h:]) # subquery needed — rate is instant vector
Existence and Counter Resets
absent(up{job="critical"}) # 1 if no series, empty if any
absent_over_time(up{job="critical"}[5m]) # missing for the entire window
resets(http_requests_total[1h]) # how many counter resets in the range
changes(up[1h]) # how many times the value changed
Use when a range-vector function needs the output of another function or aggregation:
max_over_time(rate(http_requests_total[5m])[1h:]) # peak 5m-rate over the last hour
quantile_over_time(0.99, rate(errors[1m])[1d:]) # 99th percentile of per-minute error rates over a day
deriv(rate(http_requests_total[5m])[30m:]) # is the rate accelerating?
Performance: subqueries re-run the inner expression at every resolution step. A [1d:1m] subquery runs the inner
1,440 times. For expensive inner expressions or repeated use, convert to a recording rule.
Avoid subqueries inside recording rules. Split into two recording rules.
Full syntax, alignment rules, subquery vs recording rule guidance: [${CLAUDE_SKILL_DIR}/references/subqueries.md].
Common Query Idioms
# Request rate by job
sum by (job) (rate(http_requests_total[5m]))
# Error ratio — aggregate numerator and denominator separately
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m]))
# P99 latency by handler (classic histogram)
histogram_quantile(0.99, sum by (handler, le) (rate(http_request_duration_seconds_bucket[5m])))
# Top 5 CPU consumers
topk(5, sum by (app) (rate(process_cpu_seconds_total[5m])))
# Disk-full prediction
predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[6h], 24*3600) < 0
# Alert on missing scrape target
absent(up{job="critical-service"})
# Week-over-week request rate comparison
rate(http_requests_total[5m])
-
rate(http_requests_total[5m] offset 1w)
# Batch job staleness
time() - my_batch_last_success_timestamp_seconds > 2 * 3600
Pitfalls — Quick Reference
Positive rules when writing or reviewing queries:
Apply rate() to counters only. Use delta(), idelta(), or deriv() for gauges. rate() on a gauge silently
produces nonsense.
rate() first, then aggregate.sum(rate(x[5m])), never rate(sum(x)[5m]). Counter resets are per-series.
Aggregate ratios correctly. Sum numerator and denominator separately, then divide. Never avg(ratio) or
avg(avg).
Don't average summary quantiles.avg(metric{quantile="0.95"}) is statistically invalid. Use histograms for
cross-instance aggregation.
Preserve le when aggregating classic histogram buckets.sum by (job, le) or sum without (instance).
Range ≥ 4× scrape interval, minimum 1m. Avoid rate(x[1m]) with 30s+ scrape intervals.
rate() (not irate()) for alerts.irate() is for graphing volatile counters.
without over by for most aggregations — preserves useful labels like job automatically.
offset must immediately follow the selector.sum(x offset 5m) is correct; sum(x) offset 5m is invalid.
Filter early. Push label matchers into the innermost selector.
Full pitfall catalog with diagnostics: [${CLAUDE_SKILL_DIR}/references/optimization.md].
Recording and Alerting Rules
Recording Rule Naming — level:metric:operations
instance:http_requests:rate5m # per-instance rate (no aggregation)
job:http_requests:rate5m # rolled up to per-job
job:http_errors_per_requests:ratio_rate5m # ratio, per job
job:request_latency_seconds:mean5m # average latency from Summary
level — labels remaining in the output (instance, job, cluster, instance_path, etc.)
metric — metric name; strip _total after rate()
operations — operations applied, newest first. Omit _sum when other ops present; mean replaces rate for
sum/count averages; ratio for divisions named with _per_
Recording rule examples and ratio aggregation patterns: [${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md].
Alerting Rule Essentials
-alert:HighErrorRatio# CamelCase community conventionexpr:|
sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
/sumby(job)(rate(http_requests_total[5m]))>0.05for:10m# must be true for the entire window before firingkeep_firing_for:5m# keep firing this long after expression becomes falselabels:severity:pageannotations:summary:"High error rate on {{ $labels.job }}"description:"Error ratio is {{ $value | humanizePercentage }} for {{ $labels.job }}"runbook_url:"https://wiki.example.com/runbooks/{{ $labels.job }}"
Alerting principles:
Alert on symptoms, not causes. User-visible latency, error rates — not internal-only metrics.
Page on one stack layer. If user latency is fine, don't page on a slow sub-component.
Every page must be actionable. Nothing to do → delete the alert.
Match for: to the rate window.rate(x[5m]) with for: 30s is contradictory.
Full alerting best practices, template variables, anti-patterns:
[${CLAUDE_SKILL_DIR}/references/recording-alerting-rules.md].
Application
Writing PromQL queries:
Apply conventions silently — don't narrate each rule.
Wrap counters in rate() or increase() before any further operation.
Aggregate before joining; filter at the innermost selector.
Prefer without over by when removing a noise dimension.
For histograms, prefer native when available; otherwise preserve le in aggregations.
Writing recording rules:
Follow level:metric:operations naming.
Build a hierarchy: per-instance rule first, then aggregate up. Strip _total after rate().
Aggregate ratios by computing numerator and denominator separately.
No subqueries inside recording rules — split into two rules.
Writing alerting rules:
CamelCase alert names; severity label for routing.
Alert on symptoms at the top of the stack.
Set for: to filter blips, sized to match the underlying rate window.
Use templating for context-rich notifications ({{ $labels.X }}, {{ $value }}).
Reviewing PromQL:
Cite the violation and show the fix inline.
Check first: rate() on a counter? Aggregation inside rate()? le preserved?
Integration
backend/prometheus governs instrumentation (metric types, naming, labels, exporters, cardinality). This skill governs
query writing. When both apply:
"What metric should I emit" — backend/prometheus
"How do I query this" — this skill
Naming a recording rule output — use level:metric:operations here; that becomes the metric name backend/prometheus
rules then apply to
grafana/metricsql (forthcoming) covers VictoriaMetrics' PromQL superset. MetricsQL extends PromQL with extra functions
and looser type rules — write standard PromQL when targeting either Prometheus or VictoriaMetrics; reach for
MetricsQL-only features only when the deployment is VictoriaMetrics and the standard expression is awkward.
