ops-report

name	ops-report
description	Generate a 24-hour operational health report for a PostHog service by querying Grafana dashboards and Prometheus metrics. Produces a formatted markdown report with key metrics, anomalies, and recommendations.
model	sonnet
color	green
allowed-tools	Bash, Read, Grep, Glob, Write, Edit, mcp__grafana__search_dashboards, mcp__grafana__get_dashboard_panel_queries, mcp__grafana__query_prometheus, mcp__grafana__query_prometheus_histogram, mcp__grafana__list_datasources, mcp__grafana__generate_deeplink, mcp__grafana__query_loki_logs, mcp__grafana__query_loki_stats, mcp__grafana__list_loki_label_names, mcp__grafana__list_loki_label_values, mcp__grafana-eu__search_dashboards, mcp__grafana-eu__get_dashboard_panel_queries, mcp__grafana-eu__query_prometheus, mcp__grafana-eu__query_prometheus_histogram, mcp__grafana-eu__list_datasources, mcp__grafana-eu__generate_deeplink, mcp__grafana-eu__query_loki_logs, mcp__grafana-eu__query_loki_stats, mcp__grafana-eu__list_loki_label_names, mcp__grafana-eu__list_loki_label_values
argument-hint	[service] [--window day|week|month] [--region us|eu|both]

Ops Report

Generate a 24-hour operational health report for a PostHog service by querying Grafana dashboards and Prometheus/VictoriaMetrics metrics.

Arguments (parsed from user input)

• service (optional): The service to report on (default: feature-flags). Other examples: ingestion, capture
• --window (optional): Lookback window: day (24h), week (7d), or month (30d). Default: day
• --region (optional): Grafana region to query: us, eu, or both (default: both)

Example invocations:

• /ops-report - daily report for feature flags (both regions, the default)
• /ops-report feature-flags --window week - weekly report for both regions
• /ops-report ingestion --region eu - daily ingestion report from EU only
• /ops-report --region us - US-only report

MCP Server Mapping

Region	MCP Server	Public Hostname
us	mcp__grafana__*	grafana.prod-us.posthog.dev
eu	mcp__grafana-eu__*	grafana.prod-eu.posthog.dev

When --region both (the default), run all data-gathering steps in parallel for both regions. Tag every metric, anomaly, and dashboard link with its region ([US] / [EU]) throughout.

Your Task

Follow these steps in order.

Step 1: Parse Arguments and Validate

Extract from user input:

• service - kebab-case service name, default "feature-flags"
• window - one of day, week, month. Default "day"
• region - one of us, eu, both. Default "both"

Determine which MCP servers to use:

• region=us → mcp__grafana__* only
• region=eu → mcp__grafana-eu__* only
• region=both → both mcp__grafana__* and mcp__grafana-eu__* in parallel

Map the window to query parameters:

Window	Hours	PromQL Step	Loki query range per anomaly
day	24	300s (5min)	narrow (spike +/- 15min)
week	168	1800s (30min)	wider (spike +/- 1h)
month	720	7200s (2h)	widest (spike +/- 4h)

Compute absolute UTC timestamps for the report window. All Prometheus and Loki queries must use these absolute timestamps; never use relative expressions like now-24h. Absolute timestamps ensure that spike times read from Prometheus responses correspond exactly to the report window, so Loki follow-up queries target the correct time.

For a day report generated on date D, the window is:

window_start = {D - 1 day}T00:00:00Z   (e.g. 2026-03-17T00:00:00Z)
window_end   = {D}T00:00:00Z            (e.g. 2026-03-18T00:00:00Z)

For week and month, anchor the window end to {D}T00:00:00Z and subtract accordingly:

week:  window_start = {D - 7 days}T00:00:00Z  (e.g. 2026-03-11T00:00:00Z)
month: window_start = {D - 30 days}T00:00:00Z (e.g. 2026-02-16T00:00:00Z)

Record these values:

• {window_start} and {window_end}: absolute RFC3339 timestamps for all query start/end parameters
• {window_hours}: 24, 168, or 720, for use in PromQL range selectors like [{window_hours}h]

State the computed {window_start}, {window_end}, and {window_hours} values before proceeding to Step 2.

Record the current session JSONL line count for token tracking:

claude-session-tokens --line-count

Store the output plus one as {token_baseline_line} (i.e., line_count + 1) so that --sum-from starts after the last pre-existing line.

Step 2: Discover Dashboards

Search Grafana for dashboards related to the service. For each active region, run in parallel:

mcp__grafana__search_dashboards(query="{service}")        # prod-us
mcp__grafana-eu__search_dashboards(query="{service}")     # prod-eu

Filter results to dashboards tagged with the service name or whose title contains the service name. Record each dashboard's UID, title, description, and region. Dashboard UIDs are often the same across regions; query both independently.

If no dashboards are found in either region, tell the user and stop.

Step 3: Discover Datasources

For each active region, run in parallel:

mcp__grafana__list_datasources(type="prometheus")        # prod-us
mcp__grafana-eu__list_datasources(type="prometheus")     # prod-eu

Also discover the Loki datasources for each region (needed for log queries in Step 8):

mcp__grafana__list_datasources(type="loki")        # prod-us
mcp__grafana-eu__list_datasources(type="loki")     # prod-eu

For each region, use the datasource named "VictoriaMetrics" (or the default Prometheus datasource). Record each region's Prometheus UID and Loki UID separately. Do not hardcode UIDs; discover them here.

The US Loki datasource is typically named Loki-logs (uid P44D702D3E93867EC), but always verify via discovery rather than assuming.

Step 4: Extract Key Queries from Dashboards

For the most important dashboards (the "general" or overview dashboard first, then latency, cache, and pods dashboards), extract panel queries from each active region in parallel:

mcp__grafana__get_dashboard_panel_queries(uid="{dashboard_uid}")        # prod-us
mcp__grafana-eu__get_dashboard_panel_queries(uid="{dashboard_uid}")     # prod-eu

If dashboards share the same UID across regions, the panel structure will be identical, so you only need to extract queries once and reuse them for both regions' metric queries in Step 5.

Identify the key metrics to query. Prioritize these categories:

1. Request rate - throughput over time
2. Success/error rate - 2xx/3xx vs 5xx responses
3. Latency - P50, P95, P99 percentiles
4. Resource usage - CPU, memory relative to requests/limits
5. Pod/scaling - HPA replica count, pod restarts
6. DB/cache performance - pool utilization, cache hit rates, connection times
7. Evaluation errors - service-specific error counters

Step 5: Query Metrics

Run PromQL queries against each active region's Prometheus datasource. Use range queries with the step size from the window table above:

• Use the absolute {window_start} and {window_end} timestamps computed in Step 1 as the query startTime/endTime parameters, never now-{hours}h or now
• Step size: use the value from the window mapping (300s for day, 1800s for week, 7200s for month)

Note: The ban on relative expressions applies to the query's startTime/endTime parameters. PromQL duration expressions inside the query (range selectors like [5m], [{window_hours}h], and subquery windows) remain as durations. These are lookback windows within PromQL, not the time range of the query itself.

For region=both, fire all queries for US and EU in parallel using their respective MCP servers and datasource UIDs:

mcp__grafana__query_prometheus(datasourceUid="{us_prom_uid}", ...)      # prod-us
mcp__grafana-eu__query_prometheus(datasourceUid="{eu_prom_uid}", ...)   # prod-eu

Store results separately per region so they can be compared in the report.

Important: Replace any $__rate_interval or $__interval template variables with appropriate values. For day use 5m/1m, for week use 30m/5m, for month use 2h/30m.

Step 5b: Service-Specific Capacity Checks

For certain services, query additional capacity metrics that indicate approaching limits. Run all queries for both regions in parallel using each region's respective MCP server and datasource UID.

Feature Flags

Query the number of feature flags per team to identify teams nearing the maximum allowed count. Look for relevant metrics or dashboard panels that track:

• Total flag count per team/organization
• Teams approaching the configured maximum (e.g., >80% of the limit)
• Recent growth rate in flag creation

If a direct metric isn't available, check whether the dashboard panels show this data and note any teams that appear to be nearing capacity. Flag this as an action item if any team is above 80% of the maximum.

Scheduled Task Performance

Query scheduled Celery task duration and verification fix counts from the cache dashboard. These tasks run periodically to maintain cache consistency.

Task duration (average per run, sampled over the window):

increase(posthog_celery_task_duration_seconds_sum{task_name=~"posthog\\.tasks\\.hypercache_verification\\..*|posthog\\.tasks\\.feature_flags\\.(refresh_expiring_flags_cache_entries|cleanup_stale_flags_expiry_tracking_task)|posthog\\.tasks\\.team_metadata\\.(refresh_expiring_team_metadata_cache_entries|cleanup_stale_expiry_tracking_task)|posthog\\.tasks\\.team_access_cache_tasks\\.warm_all_team_access_caches_task"}[$__rate_interval])
/
increase(posthog_celery_task_duration_seconds_count{task_name=~"posthog\\.tasks\\.hypercache_verification\\..*|posthog\\.tasks\\.feature_flags\\.(refresh_expiring_flags_cache_entries|cleanup_stale_flags_expiry_tracking_task)|posthog\\.tasks\\.team_metadata\\.(refresh_expiring_team_metadata_cache_entries|cleanup_stale_expiry_tracking_task)|posthog\\.tasks\\.team_access_cache_tasks\\.warm_all_team_access_caches_task"}[$__rate_interval])

Replace $__rate_interval per the window mapping. Query this as a range query to get a time series, then compute min, max, and average duration across the window for each task_name. Convert seconds to human-readable format (e.g., ~690s (11.5 min)).

Verification fix counts (total fixes applied over the window):

sum by(cache_type, issue_type) (increase(posthog_hypercache_verify_fixes_total[{window_hours}h]))

This tracks how many cache inconsistencies the verification tasks detected and fixed. Group by cache_type (e.g., feature_flags, team_metadata) and issue_type (e.g., cache_mismatch, missing_entry). A non-zero count is not necessarily alarming (self-healing is working), but sustained high counts or a sudden increase warrants investigation.

Task failure counts (over the window):

sum by(task_name) (increase(posthog_celery_task_failure_total{task_name=~"posthog\\.tasks\\.hypercache_verification\\..*|posthog\\.tasks\\.feature_flags\\..*|posthog\\.tasks\\.team_metadata\\..*"}[{window_hours}h]))

Any task failures should be flagged as an action item.

Task execution count (total runs over the window, reuses the duration count metric):

sum by(task_name) (increase(posthog_celery_task_duration_seconds_count{task_name=~"posthog\\.tasks\\.hypercache_verification\\..*|posthog\\.tasks\\.feature_flags\\.(refresh_expiring_flags_cache_entries|cleanup_stale_flags_expiry_tracking_task)|posthog\\.tasks\\.team_metadata\\.(refresh_expiring_team_metadata_cache_entries|cleanup_stale_expiry_tracking_task)|posthog\\.tasks\\.team_access_cache_tasks\\.warm_all_team_access_caches_task"}[{window_hours}h]))

This gives the total number of executions per task in the window. Include this as the "Runs" column in the scheduled tasks table to contextualize duration statistics.

Task retry counts (retries over the window):

sum by(task_name) (increase(posthog_celery_task_retry_total{task_name=~"posthog\\.tasks\\.hypercache_verification\\..*|posthog\\.tasks\\.feature_flags\\..*|posthog\\.tasks\\.team_metadata\\..*"}[{window_hours}h]))

Retries are typically zero. Only include them in the report when non-zero, rendered as a footnote beneath the scheduled tasks table rather than a dedicated column.

Queue health (queue depth stats and trend for feature flag queues):

Average depth over the window:

avg_over_time(posthog_celery_queue_depth{queue=~"feature_flags|feature_flags_long_running"}[{window_hours}h])

Maximum depth over the window:

max_over_time(posthog_celery_queue_depth{queue=~"feature_flags|feature_flags_long_running"}[{window_hours}h])

Trend (positive = growing, negative = draining, near-zero = stable):

deriv(posthog_celery_queue_depth{queue=~"feature_flags|feature_flags_long_running"}[{window_hours}h])

Classify the trend: deriv > 0.1 = "Growing", deriv < -0.1 = "Draining", otherwise "Stable". Queue depth is a queue-level metric, not per-task, so render it as a separate sub-section after the scheduled tasks table.

Batch refresh coverage (teams processed by the hourly batch refresh):

posthog_hypercache_teams_processed_last_run{namespace=~"feature_flags|team_metadata"}

Query this as an instant query. It shows how many teams the most recent batch refresh processed, broken down by namespace and status (success/failure). If any failures are present, flag them as an action item.

Fallback guidance: If any of the standard Celery task metrics return no data, omit that sub-section from the report rather than reporting zeros.

Sync Task Health (sync_feature_flag_last_called)

This task uses custom Prometheus metrics rather than the standard Celery task instrumentation, so it requires separate queries.

Success rate (averaged over the window):

avg_over_time(posthog_celery_sync_feature_flag_last_called_success[{window_hours}h])

Returns a 0–1 value. Multiply by 100 and report as a percentage. Values below 100% indicate failures during the window.

Duration (range query for time series):

posthog_celery_sync_feature_flag_last_called_duration_seconds

Query as a range query. Compute min, max, and average across the window. Report the average in the Avg Duration column; use min/max to inform the Assessment (e.g., flag high variance or an increasing trend). Convert to human-readable format following the same convention as other tasks (~Xs under 60s, ~Y.Z min over 60s).

Execution count (total runs in the window):

changes(posthog_celery_sync_feature_flag_last_called_duration_seconds[{window_hours}h])

Counts how many times the duration gauge changed, which corresponds to task executions. This is an approximation: if two consecutive runs produce the exact same duration, one execution may be missed.

Interpretation thresholds:

• Success rate < 100%: action item. < 90% = Warning priority, < 50% = Critical priority
• Zero executions in the window: action item (task not running)
• Duration increasing trend: note in the report for monitoring

HPA Scaling Efficiency

Query these three metrics to assess whether the HPA is tuned correctly.

% time at max replicas (fraction of the window where desired replicas hit the max):

count_over_time((kube_horizontalpodautoscaler_status_desired_replicas{horizontalpodautoscaler="posthog-feature-flags"} >= kube_horizontalpodautoscaler_spec_max_replicas{horizontalpodautoscaler="posthog-feature-flags"})[{window_hours}h:5m])
/ count_over_time(kube_horizontalpodautoscaler_status_desired_replicas{horizontalpodautoscaler="posthog-feature-flags"}[{window_hours}h:5m])

CPU headroom ratio (range query, how close the hottest pod is to the HPA target):

max(sum by (pod)(rate(container_cpu_usage_seconds_total{namespace="posthog", container="posthog-feature-flags"}[5m])) / on(pod) sum by (pod)(kube_pod_container_resource_requests{resource="cpu", namespace="posthog", container="posthog-feature-flags"})) / 0.70

Scaling events (number of HPA replica changes in the window):

changes(kube_horizontalpodautoscaler_status_desired_replicas{horizontalpodautoscaler="posthog-feature-flags"}[{window_hours}h])

Interpretation thresholds:

• % at max > 20% → action item (raise maxPods or lower CPU target)
• Headroom ratio peak > 0.9 → HPA being pushed close to scaling
• Headroom ratio peak < 0.5 → CPU target may be too conservative

Fallback: If any of these metrics return no data, omit the HPA Scaling Efficiency section from the report.

Billing Aggregator

The billing aggregator is the sole authoritative writer for flag billing counts. A silent failure here under-bills usage with no immediate symptom in latency or error rate, so it gets its own health check.

Run all queries below in parallel for each active region against that region's Prometheus datasource.

Stale-flush watchdog (instant, max across pods):

max(flags_billing_seconds_since_successful_flush)

Default flush interval is 10 seconds. Healthy pods sit well under 30s; treat anything sustained over 60s as a real problem.

Unflushed requests by cause (total over the window — this is the revenue-loss signal):

sum by(cause) (increase(flags_billing_unflushed_requests_total[{window_hours}h]))

Causes: cap_drop (memory-pressure shed at record-time), redis_error (chunk failed mid-flush; even though the requeue may succeed, the request count is booked here so the total reflects everything blocked), flush_dropped_on_error (best-effort shutdown gave up on a chunk), shutdown_drop (process exited before draining in-memory state). Any sustained non-zero total is an action item.

Flush errors by type (total over the window):

sum by(error_type) (increase(flags_billing_flush_errors_total[{window_hours}h]))

A non-zero count that does not pair with unflushed_requests_total growth means flushes are retrying successfully; pair this query with the one above before deciding severity.

Pending queue depth (max over the window):

max_over_time(sum(flags_billing_pending_records)[{window_hours}h:])

In-memory records waiting to be flushed. Should drop to ~0 each tick. Sustained growth is a leading indicator that cap_drop is imminent.

Throughput (input vs flushed-out rate, range query):

sum(rate(flags_billing_records_total[$__rate_interval]))
sum(rate(flags_billing_entries_flushed_total[$__rate_interval]))

Over a stable window the two rates should match. A widening gap (records-in > entries-flushed) means the buffer is filling faster than it drains.

Flush latency (p99 over the window):

histogram_quantile(0.99, sum by(le) (rate(flags_billing_flush_duration_ms_bucket[$__rate_interval])))

p99 trending toward the 10s tick interval means flushes risk overlapping the next tick. Pair with Pending Queue Depth — rising p99 + rising depth points at Redis as the bottleneck.

Interpretation thresholds:

• Stale-flush > 60s sustained → action item (Warning). > 120s → Critical
• Any non-zero unflushed_requests_total over the window → action item. cap_drop or shutdown_drop totals > 0 are Critical (records definitely lost). redis_error and flush_dropped_on_error are Warning by default; promote to Critical if paired with stale-flush or sustained pending growth
• Pending records depth growing trend over the window → Warning (lead time before cap_drop fires)
• Records-in / entries-flushed ratio drifting from 1.0 across the window → Warning
• Flush p99 > 5000ms sustained → Warning (half the tick interval). > 9000ms → Critical

Fallback: If flags_billing_records_total returns no data (the service may not have rolled out the aggregator yet), omit the Billing Aggregator section from the report.

Step 6: Analyze Results

For each metric time series, compute:

• Current value (most recent data point)
• Min/max over the window
• Mean over the window
• Notable spikes or dips (values more than 2x the mean, or sudden step changes)

All timestamps reported must come directly from Prometheus data point values, never from log entries, correlated signals, or inference. A log message at time T is not evidence that a metric spike occurred at time T. For each detected spike, record its peak as {spike_peak_utc} (the exact timestamp from the Prometheus data point) for use in Step 8.

Error Spike Count

After querying the 5xx error count time series, compute a spike count:

1. Count the number of data points where 5xx errors exceeded the warning threshold (thresholds apply per data point regardless of step size)
2. Classify each spike by severity:
   • Warning: 50–299 errors per data point
   • Critical: ≥ 300 errors per data point
3. Record the timestamps of the worst spikes for investigation in Step 8

Use the same spike count assessment labels as the Latency Spike Count table below.

Latency Spike Count

After querying the P99 latency time series, compute a spike count:

1. Compute the median of all P99 data points in the window
2. Set the spike threshold to max(2 × median, 200ms)
3. Count the number of sampling intervals where P99 exceeded the threshold
4. Classify each spike by severity:
   • Minor: threshold < P99 ≤ 300ms
   • Warning: 300ms < P99 ≤ 600ms
   • Critical: P99 > 600ms
5. Record the timestamps of the worst spikes for investigation in Step 8

Use these labels for the spike count assessment:

Window	None	Occasional	Elevated	Frequent
day	0	1–3	4–10	>10
week	0	1–10	11–30	>30
month	0	1–30	31–90	>90

Cross-correlate anomalies:

• Do error spikes correlate with latency spikes?
• Do latency spikes correlate with DB pool saturation?
• Do latency spikes cluster at specific times of day (e.g., peak traffic hours)?
• Do latency spikes correlate with DB pool utilization spikes?
• Do scaling events correlate with traffic surges?
• Is the HPA spending >20% of the window at max replicas? Does headroom correlate with latency?
• Are there any container restarts?
• Are any teams approaching resource or feature limits (e.g., max flag count)?
• Do task duration increases correlate with queue depth growth?
• Are batch refresh failures correlated with worker OOM kills?
• Do sync_feature_flag_last_called success rate drops correlate with worker log errors?
• Do billing aggregator flush errors or stale-flush alarms correlate with Redis latency or flag_request_redis_error spikes? Does pending queue growth precede unflushed_requests_total{cause="cap_drop"}?

For each anomaly, attempt to investigate the cause by querying additional metrics, checking for correlated events, and noting what you ruled out. The goal is to hand the reader a partially-investigated issue with clear next steps, not just a raw signal.

Step 7: Generate Dashboard Links

For each dashboard used, generate deep links with the time range for each active region in parallel. Dashboard links use relative time ranges (now-{window_hours}h) so they open correctly in the browser. This is the only place relative expressions are used:

# prod-us
mcp__grafana__generate_deeplink(
  resourceType="dashboard",
  dashboardUid="{uid}",
  timeRange={"from": "now-{window_hours}h", "to": "now"}
)

# prod-eu
mcp__grafana-eu__generate_deeplink(
  resourceType="dashboard",
  dashboardUid="{uid}",
  timeRange={"from": "now-{window_hours}h", "to": "now"}
)

Replace localhost:13000 in the generated URLs with the appropriate public Grafana hostname:

Region	Hostname
us	grafana.prod-us.posthog.dev
eu	grafana.prod-eu.posthog.dev
dev	grafana.dev.posthog.dev

Step 8: Investigate Anomalies via Loki Logs

When anomalies are detected in Step 6 (e.g., 5xx error spikes, latency spikes), query Loki access logs to investigate root causes before writing the report. This transforms "check the logs" from a next step into an already-completed investigation.

Query each region where an anomaly was detected in parallel using that region's Loki datasource UID (discovered in Step 3, not hardcoded).

Use {spike_peak_utc} from Step 6 (the actual Prometheus data point timestamp) as the centre of the Loki query window. Never substitute a time from a log message or a guess. The goal is to look at logs at the moment the metric spike occurred, not at the moment of a correlated (but possibly unrelated) log entry. Set {spike_start} = {spike_peak_utc} − 15 min and {spike_end} = {spike_peak_utc} + 15 min (or wider for week/month windows).

Discover log structure

The Contour/Envoy access logs are in each region's Loki-logs datasource (use the UID discovered in Step 3). Key labels:

• app="contour" - Envoy access logs (NOT app="envoy", which is sparse internal logs)
• upstream_cluster - The backend service, e.g., posthog_posthog-feature-flags_3001
• response_code - HTTP status code as a label (e.g., "503", "500")

Application logs use app="posthog-feature-flags" (or the service name).

Query 5xx errors

For each error spike, query the access logs in the affected region:

# prod-us
mcp__grafana__query_loki_logs(
  datasourceUid="{us_loki_uid}",
  logql='{app="contour", response_code=~"5..", upstream_cluster="posthog_posthog-feature-flags_3001"}',
  startRfc3339="{spike_start}",
  endRfc3339="{spike_end}",
  limit=20
)

# prod-eu
mcp__grafana-eu__query_loki_logs(
  datasourceUid="{eu_loki_uid}",
  logql='{app="contour", response_code=~"5..", upstream_cluster="posthog_posthog-feature-flags_3001"}',
  startRfc3339="{spike_start}",
  endRfc3339="{spike_end}",
  limit=20
)

Analyze the response_code_details field to classify errors:

• upstream_reset_before_response_started{connection_termination} - Pod scaling / connection drops
• via_upstream - Application-level error (check the app logs)
• response_timeout - Upstream took too long

Also check x_forwarded_host to identify if errors are concentrated on a single customer proxy.

Query application logs

Check whether the application itself is logging errors for each affected region:

# prod-us
mcp__grafana__query_loki_logs(
  datasourceUid="{us_loki_uid}",
  logql='{app="posthog-feature-flags"} |~ "(?i)error"',
  startRfc3339="{spike_start}",
  endRfc3339="{spike_end}",
  limit=20
)

# prod-eu
mcp__grafana-eu__query_loki_logs(
  datasourceUid="{eu_loki_uid}",
  logql='{app="posthog-feature-flags"} |~ "(?i)error"',
  startRfc3339="{spike_start}",
  endRfc3339="{spike_end}",
  limit=20
)

Broad log scan for warnings and errors (full window)

Regardless of whether anomalies were detected in Step 6, scan the application logs across the entire reporting window for warnings and errors. Run both regions in parallel:

# prod-us errors
mcp__grafana__query_loki_logs(
  datasourceUid="{us_loki_uid}",
  logql='{app="{service}"} | json | level =~ "(?i)(error|err)"',
  startRfc3339="{window_start}",
  endRfc3339="{window_end}",
  limit=50
)

# prod-us warnings
mcp__grafana__query_loki_logs(
  datasourceUid="{us_loki_uid}",
  logql='{app="{service}"} | json | level =~ "(?i)(warn|warning)"',
  startRfc3339="{window_start}",
  endRfc3339="{window_end}",
  limit=50
)

# prod-eu errors
mcp__grafana-eu__query_loki_logs(
  datasourceUid="{eu_loki_uid}",
  logql='{app="{service}"} | json | level =~ "(?i)(error|err)"',
  startRfc3339="{window_start}",
  endRfc3339="{window_end}",
  limit=50
)

# prod-eu warnings
mcp__grafana-eu__query_loki_logs(
  datasourceUid="{eu_loki_uid}",
  logql='{app="{service}"} | json | level =~ "(?i)(warn|warning)"',
  startRfc3339="{window_start}",
  endRfc3339="{window_end}",
  limit=50
)

If the json parser doesn't match (service uses unstructured logs), fall back to pattern matching:

logql='{app="{service}"} |~ "(?i)(error|err[^o])"'
logql='{app="{service}"} |~ "(?i)(warn|warning)"'

For each region, group the results by message pattern (strip timestamps, request IDs, and other variable fields) and count occurrences. Identify the top 5 most frequent warning patterns and top 5 most frequent error patterns. Note whether any patterns are new compared to what would be expected background noise.

Deduplication: If a log pattern from a spike-anchored query is already covered in the Anomalies section, do not repeat it in Warning and Error Logs. Cross-reference instead.

Sub-threshold errors: If the broad log scan finds 5xx errors but no Prometheus spike crossed the warning threshold (50 per data point), report the log errors under Warning and Error Logs but do not create an error spike anomaly or action item.

Worker task logs

Regardless of whether anomalies were detected, scan the worker logs across the entire reporting window for errors and warnings related to the service being reported on. Use the same query pattern as the broad log scan above, but with two changes:

• App label: {app="posthog-worker-django"} instead of {app="{service}"}
• Line filter: Add |= "{service_keyword}" to scope results to the service, where service_keyword is derived from service by lowercasing and replacing hyphens with underscores (e.g., feature-flags → feature_flags, ingestion → ingestion)

Run errors and warnings for both regions in parallel (4 queries total), with limit=50, using the same {window_start}/{window_end} range. Apply the same json parser with level filter, and the same unstructured-log fallback pattern if json doesn't match.

Group results by message pattern and identify the top 5 most frequent error and warning patterns, same as the broad scan. Deduplicate against the {app="{service}"} scan: if a pattern already appeared there, do not repeat it. Cross-reference worker error patterns with the sync_feature_flag_last_called success rate from Step 5b; if error logs correlate with a low success rate, note the correlation.

Step 9: Write the Report

Determine today's date from the system. The report path is:

~/dev/haacked/notes/PostHog/ops-reports/{YYYY-MM-DD}/{service}-{window}.md

For day window, the filename can omit the suffix (e.g., feature-flags.md). For week and month, include it (e.g., feature-flags-week.md).

If a report already exists at that path, tell the user and offer to overwrite it. Do not overwrite without confirmation.

Create the directory if it doesn't exist.

Use this structure for the report. The report leads with action items so the reader immediately knows what needs attention:

# {Service Name} - {Window Label} Health Report

**Date:** {YYYY-MM-DD}
**Regions:** {US (prod-us) | EU (prod-eu) | US + EU (prod-us, prod-eu)}
**Report window:** {start} to {end} UTC

## Overall Status: {Healthy | Degraded | Unhealthy}

{1-2 sentence executive summary}

## Action Items

{This section appears first so the reader immediately knows what needs attention. Each item should describe the anomaly, what investigation was already performed during report generation, and concrete next steps. If no action items exist, write "No action items. All metrics are within normal ranges."}

### {Priority}: {Action title}

- **What:** {Brief description of the anomaly or concern}
- **Evidence:** {Specific metric values, timestamps, and correlated signals}
- **Investigation so far:** {What was checked during report generation, e.g., "Correlated with deploy times - no deploys in this window" or "Error logs show timeout to downstream service X"}
- **Next steps:** {Concrete actions, e.g., "Check service X health", "Review recent deploy for regression", "Monitor for recurrence over next 24h"}

## Key Metrics Summary

When reporting on both regions, use US and EU columns so the reader can compare at a glance:

| Metric | US Current | US Range | EU Current | EU Range | Assessment |
| ------ | ---------- | -------- | ---------- | -------- | ---------- |
| ... | ... | ... | ... | ... | ... |

For single-region reports, collapse to the standard four-column format:

| Metric | Current | Range | Assessment |
| ------ | ------- | ----- | ---------- |
| ... | ... | ... | ... |

{Include an "Error spikes (5xx > 50/5min)" row after the error rate row, showing the spike count and severity breakdown. Include a "Latency spikes (P99 > {threshold}ms)" row after the P50 latency row. For dual-region, show the spike count and severity breakdown for each region separately in their respective columns. The latency {threshold} value is max(2 × median P99, 200).}

## Anomalies and Notable Events

### 1. {Event title}

{Description with timestamps and correlated metrics}

## What's Working Well

- {Bullet points of positive signals}

## Scheduled Tasks

When reporting on both regions, render a separate sub-section for each region:

### US (prod-us)

| Task | Runs | Min | Max | Avg | Fixes ({window}) |
|------|------|-----|-----|-----|------------------|
| `task_short_name` | N | ~11.5 min | ~12.1 min | ~11.8 min | 0 |

### EU (prod-eu)

| Task | Runs | Min | Max | Avg | Fixes ({window}) |
|------|------|-----|-----|-----|------------------|
| `task_short_name` | N | ~11.5 min | ~12.1 min | ~11.8 min | 0 |

{Use the short task name (e.g., `verify_and_fix_flags_cache_task`) rather than the full dotted path. For durations, use `~Xs` for values under 60s and `~Y.Z min` for values over 60s. For the Fixes column, show the total count and bold it if non-zero, appending the issue types in parentheses (e.g., **3** (cache_mismatch)). If a task had failures, note them in a row below or as a footnote. Omit tasks that had zero runs in the window. If any tasks had retries during the window, annotate the task name with an asterisk and add a footnote below the table (e.g., `*3 retries during the window`). Omit the footnote entirely when all retry counts are zero. For single-region reports, omit the sub-section headers.}

### Sync Task Health (`sync_feature_flag_last_called`)

This task uses custom metrics (not standard Celery task instrumentation).

When reporting on both regions:

| Region | Runs | Avg Duration | Success Rate | Assessment |
|--------|------|-------------|-------------|------------|
| US | N | ~Xs | XX% | Healthy / Degraded / Not Running |
| EU | N | ~Xs | XX% | Healthy / Degraded / Not Running |

{Report 0 runs as "Not Running" with a warning. Success rate < 100% should be flagged as an action item. For single-region reports, omit the Region column. Omit this sub-section entirely if the custom metrics return no data for both regions.}

### Queue Health

When reporting on both regions, show US and EU in the same table with a Region column:

| Region | Queue | Avg Depth | Max Depth | Trend |
|--------|-------|-----------|-----------|-------|
| US | `feature_flags` | N | N | Stable/Growing/Draining |
| US | `feature_flags_long_running` | N | N | Stable/Growing/Draining |
| EU | `feature_flags` | N | N | Stable/Growing/Draining |
| EU | `feature_flags_long_running` | N | N | Stable/Growing/Draining |

{Show average and maximum queue depth over the window, plus the trend derived from `deriv()`. Classify trend as "Growing" (deriv > 0.1), "Draining" (deriv < -0.1), or "Stable" (near zero). A growing queue paired with increasing task durations warrants investigation. Omit this section if queue depth metrics return no data for both regions.}

### Batch Refresh Coverage

{One-line summary per region of `posthog_hypercache_teams_processed_last_run` results, e.g., "**US:** Batch refresh processed N teams (feature_flags) and M teams (team_metadata) with no failures. **EU:** ..." If any failures are present, bold the failure count and flag as an action item. Omit this section if the metric returns no data for either region.}

## Billing Aggregator

The aggregator is the sole authoritative writer for flag billing counts; a silent failure under-bills usage without showing up in latency or error metrics.

When reporting on both regions, show US and EU in the same table with a Region column:

| Region | Signal | Value | Assessment |
|--------|--------|-------|------------|
| US | Seconds since last successful flush (max pod) | Xs | Healthy / Drifting / Stale |
| US | Unflushed requests ({window}) | N total ({cause breakdown}) | None / Warning / Critical |
| US | Flush errors ({window}) | N ({error_type breakdown}) | None / Transient / Sustained |
| US | Pending records (max / trend) | N max, Stable/Growing | Healthy / Backing up |
| US | Records-in vs entries-flushed | X.XX ratio | Matched / Diverging |
| US | Flush p99 | Xms | Healthy / Tight / At-tick |
| EU | … | … | … |

{Classify Seconds since last flush: Healthy (<30s), Drifting (30–60s), Stale (>60s). Unflushed requests: None (0), Warning (any non-zero with cause = redis_error / flush_dropped_on_error), Critical (any non-zero with cause = cap_drop / shutdown_drop — these are confirmed lost). Pending records: Healthy (drains to ~0 each tick, no trend), Backing up (sustained growth across the window). Records vs flushed ratio: Matched (0.95–1.05), Diverging (outside that band sustained). Flush p99: Healthy (<5s), Tight (5–9s), At-tick (>9s, risks overlapping next tick).}

{If any signal lands in a non-healthy band, add a corresponding action item to the top of the report describing the signal, the cause/error_type breakdown, and the next step (typically: check Redis health, then inspect aggregator pod logs at the spike time).}

{Omit this entire section if `flags_billing_records_total` returned no data for both regions.}

## HPA Scaling Efficiency

When reporting on both regions, show US and EU in the same table with a Region column:

| Region | Metric | Value | Assessment |
|--------|--------|-------|------------|
| US | Time at max replicas | X% | OK / Elevated / Critical |
| US | CPU headroom (max pod / target) | X.XX avg, X.XX peak | Comfortable / Tight / Over-target |
| US | Scaling events | N | Stable / Moderate / Volatile |
| EU | Time at max replicas | X% | OK / Elevated / Critical |
| EU | CPU headroom (max pod / target) | X.XX avg, X.XX peak | Comfortable / Tight / Over-target |
| EU | Scaling events | N | Stable / Moderate / Volatile |

## Warning and Error Logs

Summary of warning and error log messages observed across the full reporting window.

### Errors

When reporting on both regions, use sub-sections per region. For each region, list the top recurring error patterns in a table:

| Count | Message Pattern | First Seen | Last Seen |
|-------|----------------|------------|-----------|
| N | `short description of the error pattern` | HH:MM UTC | HH:MM UTC |

If no errors were logged, write: "No error-level log messages observed in this window."

### Warnings

| Count | Message Pattern | First Seen | Last Seen |
|-------|----------------|------------|-----------|
| N | `short description of the warning pattern` | HH:MM UTC | HH:MM UTC |

If no warnings were logged, write: "No warning-level log messages observed in this window."

{Cross-reference any log patterns against the anomalies identified in Step 6. If a log pattern correlates with a metric spike, note it here and link to the relevant anomaly section. Promote recurring or high-volume error patterns to the Action Items section if they warrant investigation.}

### Worker Task Logs

Summary of service-related error and warning messages from `posthog-worker-django` across the full reporting window. When reporting on both regions, use sub-sections per region (same as Errors and Warnings above).

| Count | Message Pattern | First Seen | Last Seen |
|-------|----------------|------------|-----------|
| N | `short description of the worker error pattern` | HH:MM UTC | HH:MM UTC |

If no worker task errors or warnings were logged, write: "No service-related worker task log messages observed in this window."

{Cross-reference worker error patterns against the `sync_feature_flag_last_called` success rate from the Scheduled Tasks section. If a worker error pattern correlates with a low success rate, note it and link to the relevant section. Promote high-volume worker errors to Action Items if they indicate a systemic issue. Only omit this sub-section if worker log queries could not be run or no worker log datasource is available for both regions; if queries ran successfully but returned no messages, include this sub-section with the "No service-related worker task log messages observed in this window." sentence.}

## {Service-specific sections as appropriate}

{e.g., Cache Performance, DB Connection Pool, Capacity/Limits, etc.}

## Dashboard Links

These links require VPN access and Cognito authentication:

### US (prod-us)

- [Dashboard Name](grafana.prod-us.posthog.dev/...)

### EU (prod-eu)

- [Dashboard Name](grafana.prod-eu.posthog.dev/...)

## Data Sources

{Brief description of how the data was collected}

Step 10: Token Usage, Lint, and Confirm

Token Usage

Query the token usage for this report:

claude-session-tokens --sum-from {token_baseline_line}

Parse the JSON output. Store {report_tokens} (the total_tokens value) and {report_output_tokens} (the output_tokens value) for use in the Data Sources section and Slack summary. The count is approximate (excludes the final message that writes the report).

Append the following line to the Data Sources section of the report:

**Report token usage (approximate):** ~{report_tokens} total ({report_output_tokens} output). Excludes the final report-writing message.

Lint

Run markdownlint on the report if available:

npx markdownlint-cli {report_path} 2>&1 || true

Fix any lint errors. Then tell the user where the report was saved and offer a brief summary of the findings.

Step 11: Copy Slack Summary

After saving the report, generate a condensed Slack summary in HTML format and copy it to the clipboard.

HTML Structure

Build an HTML snippet containing:

1. Overall status with the executive summary
2. Action items (if any), each as a list item
3. Key metrics (1-2 lines of the most important numbers)

Use the same HTML conventions as the standup skill:

<p><b>{Service Name} {Window} Health: {Healthy | Degraded | Unhealthy}</b></p>
<p>{Executive summary sentence}</p>
<p><b>Action Items:</b></p>
<ul>
<li>[{Priority}] {Action title}: {Brief description}</li>
</ul>
<p><b>Key Metrics:</b></p>
<ul>
<li>Request rate: {value} | Error rate: {value} | P99 latency: {value}</li>
</ul>
<p><b>Tokens:</b> ~{report_tokens}</p>
<p>Full report: <code>{report_path}</code></p>

If there are no action items, replace the Action Items section with:

<p>No action items. All metrics within normal ranges.</p>

Copy to Clipboard

swift ~/bin/copy-html-to-clipboard.swift <<'EOF'
{generated HTML}
EOF

Display: "Copied Slack summary to clipboard. Paste directly into Slack!"

Assessment Criteria

Use these thresholds to determine the overall status:

Status	Criteria
Healthy	Success rate >99%, P99 <500ms, no sustained error spikes, no restarts, no latency spikes
Degraded	Success rate 95-99%, P99 500ms-2s, brief error spikes, scaling pressure, or occasional minor latency spikes
Unhealthy	Success rate <95%, P99 >2s, sustained errors, restarts, pool exhaustion, frequent latency spikes, or any critical spikes

Writing Style

• Be factual and specific with numbers and timestamps
• Distinguish between transient blips (single data points) and sustained issues
• Note boundary artifacts (e.g., increase() at query range boundaries producing inflated first values)
• Cross-reference metrics to establish causation, not just correlation
• Lead with action items; the reader should know within 10 seconds whether the report needs their attention
• For each action item, document what investigation was already performed and what remains
• Keep next steps actionable and tied to specific observations
• Use UTC timestamps throughout
• Never use em dashes. Use commas, colons, parentheses, or periods instead

What You Do NOT Do

• Guess at metric values without querying
• Report a metric spike at a timestamp read from a log entry instead of from the Prometheus data point
• Report on dashboards that don't exist for the service
• Create empty directories
• Use localhost URLs in the report
• Alarm on known benign patterns (e.g., diurnal traffic drops)