promql-generator

# PromQL Query Generator ## Overview This skill provides a comprehensive, interactive workflow for generating production-ready PromQL queries with best practices built-in. Generate queries for monitoring dashboards, alerting rules, and ad-hoc analysis with an emphasis on user collaboration and planning before code generation. ## When to Use This Skill Invoke this skill when: - Creating new PromQL queries from scratch - Building monitoring dashboards (Grafana, Prometheus UI, etc.) - Implementing alerting rules for Prometheus Alertmanager - Analyzing metrics for troubleshooting or capacity planning - Converting monitoring requirements into PromQL expressions - Learning PromQL or teaching others - The user asks to "create", "generate", "build", or "write" PromQL queries - Working with Prometheus metrics (counters, gauges, histograms, summaries) - Implementing RED (Rate, Errors, Duration) or USE (Utilization, Saturation, Errors) metrics ## Interactive Query Planning Workflow **CRITICAL**: This skill emphasizes **interactive planning** before query generation. Always engage the user in a collaborative planning process to ensure the generated query matches their exact intentions. Follow this workflow when generating PromQL queries: ### Stage 1: Understand the Monitoring Goal Start by understanding what the user wants to monitor or measure. Ask clarifying questions to gather requirements: 1. **Primary Goal**: What are you trying to monitor or measure? - Request rate (requests per second) - Error rate (percentage of failed requests) - Latency/duration (response times, percentiles) - Resource usage (CPU, memory, disk, network) - Availability/uptime - Queue depth, saturation, throughput - Custom business metrics 2. **Use Case**: What will this query be used for? - Dashboard visualization (Grafana, Prometheus UI) - Alerting rule (firing when threshold exceeded) - Ad-hoc troubleshooting/analysis - Recording rule (pre-computed aggregation) - Capacity planning or SLO tracking 3. **Context**: Any additional context? - Service/application name - Team or project - Priority level - Existing metrics or naming conventions Use the **AskUserQuestion** tool to gather this information if not provided. > **When to Ask vs. Infer**: If the user's initial request already clearly specifies the goal, use case, and context (e.g., "Create an alert for P95 latency > 500ms for payment-service"), you may acknowledge these details in your response instead of re-asking. Only ask clarifying questions for information that is missing or ambiguous. ### Stage 2: Identify Available Metrics Determine which metrics are available and relevant: 1. **Metric Discovery**: What metrics are available? - Ask the user for metric names - If uncertain, suggest common naming patterns - Check for metric type indicators in the name: - `_total` suffix → Counter - `_bucket`, `_sum`, `_count` suffix → Histogram - No suffix → Likely Gauge - `_created` suffix → Counter creation timestamp 2. **Metric Type Identification**: Confirm the metric type(s) - **Counter**: Cumulative metric that only increases (or resets to zero) - Examples: `http_requests_total`, `errors_total`, `bytes_sent_total` - Use with: `rate()`, `irate()`, `increase()` - **Gauge**: Point-in-time value that can go up or down - Examples: `memory_usage_bytes`, `cpu_temperature_celsius`, `queue_length` - Use with: `avg_over_time()`, `min_over_time()`, `max_over_time()`, or directly - **Histogram**: Buckets of observations with cumulative counts - Examples: `http_request_duration_seconds_bucket`, `response_size_bytes_bucket` - Use with: `histogram_quantile()`, `rate()` - **Summary**: Pre-calculated quantiles with count and sum - Examples: `rpc_duration_seconds{quantile="0.95"}` - Use `_sum` and `_count` for averages; don't average quantiles 3. **Label Discovery**: What labels are available on these metrics? - Common labels: `job`, `instance`, `environment`, `service`, `endpoint`, `status_code`, `method` - Ask which labels are important for filtering or grouping Use the **AskUserQuestion** tool to confirm metric names, types, and available labels. ### Stage 3: Determine Query Parameters Gather specific requirements for the query. #### Pre-confirmation for User-Provided Parameters **IMPORTANT**: When the user has already specified parameters in their initial request (e.g., "5-minute window", "500ms threshold", "> 5% error rate"), you MUST: 1. **Acknowledge the provided values** explicitly in your response 2. **Present them as pre-filled defaults** in AskUserQuestion with the first option being "Use specified values" 3. **Allow quick confirmation** rather than re-asking for information already given **Example**: If user says "alert when P95 latency exceeds 500ms", use: ``` AskUserQuestion: - Question: "Confirm the alert threshold?" - Options: 1. "500ms (as specified)" - Use the threshold from your request 2. "Different threshold" - Let me specify a different value ``` This respects the user's input and speeds up the workflow while still allowing modifications. 1. **Time Range**: What time window should the query cover? - Instant value (current) - Rate over time (`[5m]`, `[1h]`, `[1d]`) - For rate calculations: typically `[1m]` to `[5m]` for real-time, `[1h]` to `[1d]` for trends - Rule of thumb: Rate range should be at least 4x the scrape interval 2. **Label Filtering**: Which labels should filter the data? - Exact matches: `job="api-server"`, `status_code="200"` - Negative matches: `status_code!="200"` - Regex matches: `instance=~"prod-.*"` - Multiple conditions: `{job="api", environment="production"}` 3. **Aggregation**: Should the data be aggregated? - **No aggregation**: Return all time series as-is - **Aggregate by labels**: `sum by (job, endpoint)`, `avg by (instance)` - **Aggregate without labels**: `sum without (instance, pod)`, `avg without (job)` - Common aggregations: `sum`, `avg`, `max`, `min`, `count`, `topk`, `bottomk` 4. **Thresholds or Conditions**: Are there specific conditions? - For alerting: threshold values (e.g., error rate > 5%) - For filtering: only show series above/below a value - For comparison: compare against historical data (offset) Use the **AskUserQuestion** tool to gather or confirm these parameters. When the user has already provided values (e.g., "5-minute window", "> 5%"), present them as the default option for confirmation. ### Stage 4: Present the Query Plan **BEFORE GENERATING ANY CODE**, present a plain-English query plan and ask for user confirmation: ``` ## PromQL Query Plan Based on your requirements, here's what the query will do: **Goal**: [Describe the monitoring goal in plain English] **Query Structure**: 1. Start with metric: `[metric_name]` 2. Filter by labels: `{label1="value1", label2="value2"}` 3. Apply function: `[function_name]([metric][time_range])` 4. Aggregate: `[aggregation] by ([label_list])` 5. Additional operations: [any calculations, ratios, or transformations] **Expected Output**: - Data type: [instant vector/scalar] - Labels in result: [list of labels] - Value represents: [what the number means] - Typical range: [expected value range] **Example Interpretation**: If the query returns `0.05`, it means: [plain English explanation] **Does this match your intentions?** - If yes, I'll generate the query and validate it - If no, let me know what needs to change ``` Use the **AskUserQuestion** tool to confirm the plan with options: - "Yes, generate this query" - "Modify [specific aspect]" - "Show me alternative approaches" When the user chooses: - **"Modify [specific aspect]"**: ask one focused follow-up question about what to change (metric, labels, function, time range, threshold, or output shape), then present an updated plan before generating. - **"Show me alternative approaches"**: provide at least two valid query plans with trade-offs (accuracy, cost, cardinality, readability), then ask the user to choose one before generating. ### Stage 5: Generate the PromQL Query Once the user confirms the plan, generate the actual PromQL query following best practices. #### IMPORTANT: Consult Reference Files Before Generating **Before writing any query code**, you MUST: 1. **Identify the query category** first (histogram, RED, USE, function-specific, optimization, etc.). 2. **Read only the relevant reference section(s)** using the Read tool: - For histogram queries → Read `references/metric_types.md` (Histogram section) - For error/latency patterns → Read `references/promql_patterns.md` (RED method section) - For resource monitoring → Read `references/promql_patterns.md` (USE method section) - For optimization questions → Read `references/best_practices.md` - For specific functions → Read `references/promql_functions.md` - Re-read a section only if requirements changed or you have not consulted it yet in the current thread. 3. **If a needed reference cannot be read**, state the issue and continue with best-effort generation using the most applicable documented pattern you already have. 4. **Cite the applicable pattern or best practice** in your response: ``` As documented in references/promql_patterns.md (Pattern 3: Latency Percentile): # 95th percentile latency histogram_quantile(0.95, sum by (le) (rate(...))) ``` 5. **Reference example files** when generating similar queries: ``` Based on examples/red_method.promql (lines 64-82): # P95 latency with proper histogram_quantile usage ``` This keeps generated queries aligned with documented patterns while avoiding unnecessary full-file rereads on iterative follow-ups. #### Best Practices for Query Generation 1. **Always Use Label Filters** ```promql # Good: Specific filtering reduces cardinality rate(http_requests_total{job="api-server", environment="prod"}[5m]) # Bad: Matches all time series, high cardinality rate(http_requests_total[5m]) ``` 2. **Use Appropriate Functions for Metric Types** ```promql # Counter: Use rate() or increase() rate(http_requests_total[5m]) # Gauge: Use directly or with *_over_time() memory_usage_bytes avg_over_time(memory_usage_bytes[5m]) # Histogram: Use histogram_quantile() histogram_quantile(0.95, sum by (le) (rate(http_request_duration_seconds_bucket[5m])) ) ``` 3. **Apply Aggregations with by() or without()** ```promql # Aggregate by specific labels (keeps only these labels) sum by (job, endpoint) (rate(http_requests_total[5m])) # Aggregate without specific labels (removes these labels) sum without (instance, pod) (rate(http_requests_total[5m])) ``` 4. **Use Exact Matches Over Regex When Possible** ```promql # Good: Faster exact match http_requests_total{status_code="200"} # Bad: Slower regex match when not needed http_requests_total{status_code=~"200"} ``` 5. **Calculate Ratios Properly** ```promql # Error rate: errors / total requests sum(rate(http_requests_total{status_code=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) ``` 6. **Use Recording Rules for Complex Queries** - If a query is used frequently or is computationally expensive - Pre-aggregate data to reduce query load - Follow naming convention: `level:metric:operations` 7. **Format for Readability** ```promql # Good: Multi-line for complex queries histogram_quantile(0.95, sum by (le, job) ( rate(http_request_duration_seconds_bucket{job="api-server"}[5m]) ) ) ``` #### Common Query Patterns **Pattern 1: Request Rate** ```promql # Requests per second rate(http_requests_total{job="api-server"}[5m]) # Total requests per second across all instances sum(rate(http_requests_total{job="api-server"}[5m])) ``` **Pattern 2: Error Rate** ```promql # Error ratio (0 to 1) sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m])) / sum(rate(http_requests_total{job="api-server"}[5m])) # Error percentage (0 to 100) ( sum(rate(http_requests_total{job="api-server", status_code=~"5.."}[5m])) / sum(rate(http_requests_total{job="api-server"}[5m])) ) * 100 ``` **Pattern 3: Latency Percentile (Histogram)** ```promql # 95th percentile latency histogram_quantile(0.95, sum by (le) ( rate(http_request_duration_seconds_bucket{job="api-server"}[5m]) ) ) ``` **Pattern 4: Resource Usage** ```promql # Current memory usage process_resident_memory_bytes{job="api-server"} # Average CPU usage over 5 minutes avg_over_time(process_cpu_seconds_total{job="api-server"}[5m]) ``` **Pattern 5: Availability** ```promql # Percentage of up instances ( count(up{job="api-server"} == 1) / count(up{job="api-server"}) ) * 100 ``` **Pattern 6: Saturation/Queue Depth** ```promql # Average queue length avg_over_time(queue_depth{job="worker"}[5m]) # Maximum queue depth in the last hour max_over_time(queue_depth{job="worker"}[1h]) ``` ### Stage 6: Validate the Generated Query **ALWAYS attempt to validate the generated query first** using the devops-skills:promql-validator skill: ``` After generating the query, automatically invoke: Skill(devops-skills:promql-validator) The devops-skills:promql-validator skill will: 1. Check syntax correctness 2. Validate semantic logic (correct functions for metric types) 3. Identify anti-patterns and inefficiencies 4. Suggest optimizations 5. Explain what the query does 6. Verify it matches user intent ``` **Validation checklist**: - Syntax is correct (balanced brackets, valid operators) - Metric type matches function usage - Label filters are specific enough - Aggregation is appropriate - Time ranges are reasonable - No known anti-patterns