promql-generator

name	promql-generator
description	Generate/create/write PromQL queries, metric expressions, alerting rules, recording rules, Prometheus dashboards. Use when this capability is needed.
metadata	{"author":"akin-ozer"}

PromQL Query Generator

Overview

This skill provides a comprehensive, interactive workflow for generating production-ready PromQL queries with best practices built-in. Generate queries for monitoring dashboards, alerting rules, and ad-hoc analysis with an emphasis on user collaboration and planning before code generation.

When to Use This Skill

Invoke this skill when:

• Creating new PromQL queries from scratch
• Building monitoring dashboards (Grafana, Prometheus UI, etc.)
• Implementing alerting rules for Prometheus Alertmanager
• Analyzing metrics for troubleshooting or capacity planning
• Converting monitoring requirements into PromQL expressions
• Learning PromQL or teaching others
• The user asks to "create", "generate", "build", or "write" PromQL queries
• Working with Prometheus metrics (counters, gauges, histograms, summaries)
• Implementing RED (Rate, Errors, Duration) or USE (Utilization, Saturation, Errors) metrics

Interactive Query Planning Workflow

CRITICAL: This skill emphasizes interactive planning before query generation. Always engage the user in a collaborative planning process to ensure the generated query matches their exact intentions.

Follow this workflow when generating PromQL queries:

Stage 1: Understand the Monitoring Goal

Start by understanding what the user wants to monitor or measure. Ask clarifying questions to gather requirements:

1. Primary Goal: What are you trying to monitor or measure?

   • Request rate (requests per second)
   • Error rate (percentage of failed requests)
   • Latency/duration (response times, percentiles)
   • Resource usage (CPU, memory, disk, network)
   • Availability/uptime
   • Queue depth, saturation, throughput
   • Custom business metrics

2. Use Case: What will this query be used for?

   • Dashboard visualization (Grafana, Prometheus UI)
   • Alerting rule (firing when threshold exceeded)
   • Ad-hoc troubleshooting/analysis
   • Recording rule (pre-computed aggregation)
   • Capacity planning or SLO tracking

3. Context: Any additional context?

   • Service/application name
   • Team or project
   • Priority level
   • Existing metrics or naming conventions

Use the AskUserQuestion tool to gather this information if not provided.

When to Ask vs. Infer: If the user's initial request already clearly specifies the goal, use case, and context (e.g., "Create an alert for P95 latency > 500ms for payment-service"), you may acknowledge these details in your response instead of re-asking. Only ask clarifying questions for information that is missing or ambiguous.

Stage 2: Identify Available Metrics

Determine which metrics are available and relevant:

1. Metric Discovery: What metrics are available?

   • Ask the user for metric names
   • If uncertain, suggest common naming patterns
   • Check for metric type indicators in the name:
     • _total suffix → Counter
     • _bucket, _sum, _count suffix → Histogram
     • No suffix → Likely Gauge
     • _created suffix → Counter creation timestamp

2. Metric Type Identification: Confirm the metric type(s)

   • Counter: Cumulative metric that only increases (or resets to zero)
     • Examples: http_requests_total, errors_total, bytes_sent_total
     • Use with: rate(), irate(), increase()
   • Gauge: Point-in-time value that can go up or down
     • Examples: memory_usage_bytes, cpu_temperature_celsius, queue_length
     • Use with: avg_over_time(), min_over_time(), max_over_time(), or directly
   • Histogram: Buckets of observations with cumulative counts
     • Examples: http_request_duration_seconds_bucket, response_size_bytes_bucket
     • Use with: histogram_quantile(), rate()
   • Summary: Pre-calculated quantiles with count and sum
     • Examples: rpc_duration_seconds{quantile="0.95"}
     • Use _sum and _count for averages; don't average quantiles

3. Label Discovery: What labels are available on these metrics?

   • Common labels: job, instance, environment, service, endpoint, status_code, method
   • Ask which labels are important for filtering or grouping

Use the AskUserQuestion tool to confirm metric names, types, and available labels.

Stage 3: Determine Query Parameters

Gather specific requirements for the query.

Pre-confirmation for User-Provided Parameters

IMPORTANT: When the user has already specified parameters in their initial request (e.g., "5-minute window", "500ms threshold", "> 5% error rate"), you MUST:

1. Acknowledge the provided values explicitly in your response
2. Present them as pre-filled defaults in AskUserQuestion with the first option being "Use specified values"
3. Allow quick confirmation rather than re-asking for information already given

Example: If user says "alert when P95 latency exceeds 500ms", use:

AskUserQuestion:
- Question: "Confirm the alert threshold?"
- Options:
  1. "500ms (as specified)" - Use the threshold from your request
  2. "Different threshold" - Let me specify a different value

This respects the user's input and speeds up the workflow while still allowing modifications.

1. Time Range: What time window should the query cover?

   • Instant value (current)
   • Rate over time ([5m], [1h], [1d])
   • For rate calculations: typically [1m] to [5m] for real-time, [1h] to [1d] for trends
   • Rule of thumb: Rate range should be at least 4x the scrape interval

2. Label Filtering: Which labels should filter the data?

   • Exact matches: job="api-server", status_code="200"
   • Negative matches: status_code!="200"
   • Regex matches: instance=~"prod-.*"
   • Multiple conditions: {job="api", environment="production"}

3. Aggregation: Should the data be aggregated?

   • No aggregation: Return all time series as-is
   • Aggregate by labels: sum by (job, endpoint), avg by (instance)
   • Aggregate without labels: sum without (instance, pod), avg without (job)
   • Common aggregations: sum, avg, max, min, count, topk, bottomk

4. Thresholds or Conditions: Are there specific conditions?

   • For alerting: threshold values (e.g., error rate > 5%)
   • For filtering: only show series above/below a value
   • For comparison: compare against historical data (offset)

Use the AskUserQuestion tool to gather or confirm these parameters. When the user has already provided values (e.g., "5-minute window", "> 5%"), present them as the default option for confirmation.

Stage 4: Present the Query Plan

BEFORE GENERATING ANY CODE, present a plain-English query plan and ask for user confirmation:

## PromQL Query Plan

Based on your requirements, here's what the query will do:

**Goal**: [Describe the monitoring goal in plain English]

**Query Structure**:
1. Start with metric: `[metric_name]`
2. Filter by labels: `{label1="value1", label2="value2"}`
3. Apply function: `[function_name]([metric][time_range])`
4. Aggregate: `[aggregation] by ([label_list])`
5. Additional operations: [any calculations, ratios, or transformations]

**Expected Output**:
- Data type: [instant vector/scalar]
- Labels in result: [list of labels]
- Value represents: [what the number means]
- Typical range: [expected value range]

**Example Interpretation**:
If the query returns `0.05`, it means: [plain English explanation]

**Does this match your intentions?**
- If yes, I'll generate the query and validate it
- If no, let me know what needs to change

Use the AskUserQuestion tool to confirm the plan with options:

• "Yes, generate this query"
• "Modify [specific aspect]"
• "Show me alternative approaches"

When the user chooses:

• "Modify [specific aspect]": ask one focused follow-up question about what to change (metric, labels, function, time range, threshold, or output shape), then present an updated plan before generating.
• "Show me alternative approaches": provide at least two valid query plans with trade-offs (accuracy, cost, cardinality, readability), then ask the user to choose one before generating.

Stage 5: Generate the PromQL Query

Once the user confirms the plan, generate the actual PromQL query following best practices.

IMPORTANT: Consult Reference Files Before Generating

Before writing any query code, you MUST:

1. Identify the query category first (histogram, RED, USE, function-specific, optimization, etc.).

2. Read only the relevant reference section(s) using the Read tool:

   • For histogram queries → Read references/metric_types.md (Histogram section)
   • For error/latency patterns → Read references/promql_patterns.md (RED method section)
   • For resource monitoring → Read references/promql_patterns.md (USE method section)
   • For optimization questions → Read references/best_practices.md
   • For specific functions → Read references/promql_functions.md
   • Re-read a section only if requirements changed or you have not consulted it yet in the current thread.

3. If a needed reference cannot be read, state the issue and continue with best-effort generation using the most applicable documented pattern you already have.

4. Cite the applicable pattern or best practice in your response:

   As documented in references/promql_patterns.md (Pattern 3: Latency Percentile):
   # 95th percentile latency
   histogram_quantile(0.95, sum by (le) (rate(...)))
   

5. Reference example files when generating similar queries:

   Based on examples/red_method.promql (lines 64-82):
   # P95 latency with proper histogram_quantile usage
   

This keeps generated queries aligned with documented patterns while avoiding unnecessary full-file rereads on iterative follow-ups.

Best Practices for Query Generation

1. Always Use Label Filters

   # Good: Specific filtering reduces cardinality
   rate(http_requests_total{job="api-server", environment="prod"}[5m])
   
   # Bad: Matches all time series, high cardinality
   rate(http_requests_total[5m])
   

2. Use Appropriate Functions for Metric Types

   # Counter: Use rate() or increase()
   rate(http_requests_total[5m])
   
   # Gauge: Use directly or with *_over_time()
   memory_usage_bytes
   avg_over_time(memory_usage_bytes[5m])
   
   # Histogram: Use histogram_quantile()
   histogram_quantile(0.95,
     sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
   )
   

3. Apply Aggregations with by() or without()

   # Aggregate by specific labels (keeps only these labels)
   sum by (job, endpoint) (rate(http_requests_total[5m]))
   
   # Aggregate without specific labels (removes these labels)
   sum without (instance, pod) (rate(http_requests_total[5m]))
   

4. Use Exact Matches Over Regex When Possible

   # Good: Faster exact match
   http_requests_total{status_code="200"}
   
   # Bad: Slower regex match when not needed
   http_requests_total{status_code=~"200"}
   

5. Calculate Ratios Properly

   # Error rate: errors / total requests
   sum(rate(http_requests_total{status_code=~"5.."}[5m]))
   /
   sum(rate(http_requests_total[5m]))
   

6. Use Recording Rules for Complex Queries

   • If a query is used frequently or is computationally expensive
   • Pre-aggregate data to reduce query load
   • Follow naming convention: level:metric:operations

7. Format for Readability

   # Good: Multi-line for complex queries
   histogram_quantile(0.95,
     sum by (le, job) (
       rate(http_request_duration_seconds_bucket{job="api-server"}[5m])
     )
   )
   

Common Query Patterns

Pattern 1: Request Rate

# Requests per second
rate(http_requests_total{job="api-server"}[5m])

# Total requests per second across all instances
sum(rate(http_requests_total{job="api-server"}[5m]))

Pattern 2: Error Rate

# Error ratio (0 to 1)
sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]))
/
sum(rate(http_requests_total{job="api-server"}[5m]))

# Error percentage (0 to 100)
(
  sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m]))
  /
  sum(rate(http_requests_total{job="api-server"}[5m]))
) * 100

Pattern 3: Latency Percentile (Histogram)

# 95th percentile latency
histogram_quantile(0.95,
  sum by (le) (
    rate(http_request_duration_seconds_bucket{job="api-server"}[5m])
  )
)

Pattern 4: Resource Usage

# Current memory usage
process_resident_memory_bytes{job="api-server"}

# Average CPU usage over 5 minutes
avg_over_time(process_cpu_seconds_total{job="api-server"}[5m])

Pattern 5: Availability

# Percentage of up instances
(
  count(up{job="api-server"} == 1)
  /
  count(up{job="api-server"})
) * 100

Pattern 6: Saturation/Queue Depth

# Average queue length
avg_over_time(queue_depth{job="worker"}[5m])

# Maximum queue depth in the last hour
max_over_time(queue_depth{job="worker"}[1h])

Stage 6: Validate the Generated Query

ALWAYS attempt to validate the generated query first using the devops-skills:promql-validator skill:

After generating the query, automatically invoke:
Skill(devops-skills:promql-validator)

The devops-skills:promql-validator skill will:
1. Check syntax correctness
2. Validate semantic logic (correct functions for metric types)
3. Identify anti-patterns and inefficiencies
4. Suggest optimizations
5. Explain what the query does
6. Verify it matches user intent

Validation checklist:

• Syntax is correct (balanced brackets, valid operators)
• Metric type matches function usage
• Label filters are specific enough
• Aggregation is appropriate
• Time ranges are reasonable
• No known anti-patterns