observability-testing-patterns

# Observability Testing Patterns ## Browser engine Dashboard screenshot validation and alert-UI verification go through the **qe-browser** fleet skill (`.claude/skills/qe-browser/`). Vibium is installed by `aqe init`. Typical dashboard regression workflow: ```bash vibium go "$GRAFANA_URL/d/api-latency" vibium wait load node .claude/skills/qe-browser/scripts/assert.js --checks '[ {"kind": "selector_visible", "selector": ".panel-title"}, {"kind": "no_console_errors"}, {"kind": "no_failed_requests"}, {"kind": "element_count", "selector": ".panel", "op": ">=", "count": 4} ]' node .claude/skills/qe-browser/scripts/visual-diff.js --name "grafana-api-latency" ``` <default_to_action> When testing observability infrastructure, dashboards, or monitoring: 1. VALIDATE data accuracy (source data matches what the dashboard displays) 2. TEST alert rules fire correctly at defined thresholds 3. VERIFY log aggregation completeness (no missing logs across services) 4. TRACE distributed requests end-to-end through APM 5. MEASURE dashboard performance (render time, query latency) 6. CONFIRM SLA/SLO compliance through synthetic monitoring 7. TEST metric pipeline integrity from collection to display **Quick Pattern Selection:** - Dashboard shows wrong numbers -> Data accuracy validation - Alerts not firing -> Alert rule threshold testing - Missing logs in Kibana -> Log aggregation completeness - Slow dashboard -> Dashboard performance testing - Broken traces -> APM trace validation - SLA disputes -> SLO compliance validation **Critical Success Factors:** - Observability is only as good as the data it shows - A dashboard that lies is worse than no dashboard - Alert fatigue kills response times; test thresholds carefully </default_to_action> ## Quick Reference Card ### When to Use - Validating dashboard data accuracy (Kibana, Grafana, Datadog) - Testing alert rule thresholds and notification delivery - Verifying log aggregation completeness across microservices - Validating distributed tracing (APM) correctness - Measuring SLA/SLO compliance - Testing metric pipeline integrity (collection -> aggregation -> display) ### Testing Levels | Level | Purpose | Dependencies | Speed | |-------|---------|--------------|-------| | Query Validation | Elasticsearch/PromQL query accuracy | Data source | Fast | | Dashboard Accuracy | Visual matches source data | Full stack | Medium | | Alert Threshold | Trigger and notification testing | Alerting stack | Medium | | Pipeline Integrity | End-to-end metric flow | Full pipeline | Slower | | Performance | Dashboard render time, query latency | Full stack | Slower | ### Critical Test Scenarios | Scenario | Must Test | Example | |----------|----------|---------| | Data Accuracy | Dashboard = source truth | Order count on dashboard = DB count | | Alert Firing | Threshold triggers alert | Error rate > 5% fires PagerDuty | | Alert Recovery | Auto-resolve when recovered | Error rate drops below 5% clears alert | | Log Completeness | All services emit logs | 10 microservices, all logs in Kibana | | Trace Integrity | Full request path visible | Auth -> API -> DB -> Cache spans | | SLO Compliance | Error budget tracking | 99.9% availability over 30 days | | Time Accuracy | Timestamps aligned | Log timestamp matches event time | ### Tools - **Dashboards**: Kibana, Grafana, Datadog, New Relic - **Search**: Elasticsearch, OpenSearch, Loki - **Metrics**: Prometheus, InfluxDB, CloudWatch - **Tracing**: Jaeger, Zipkin, Datadog APM, OpenTelemetry - **Alerting**: PagerDuty, OpsGenie, Alertmanager - **Synthetic**: Datadog Synthetics, Checkly, Playwright ### Agent Coordination - `qe-integration-tester`: Validate data pipelines, query accuracy, log completeness - `qe-performance-tester`: Dashboard render performance, query latency - `qe-visual-tester`: Dashboard visual regression, layout accuracy --- ## Dashboard Data Accuracy Validation ### Compare Source Data to Dashboard ```javascript describe('Dashboard Data Accuracy', () => { it('order count on dashboard matches database', async () => { // Step 1: Get ground truth from source database const dbResult = await db.query( "SELECT COUNT(*) as count FROM orders WHERE created_at >= NOW() - INTERVAL '24 HOURS'" ); const dbCount = parseInt(dbResult.rows[0].count); // Step 2: Query Elasticsearch (same data source as dashboard) const esResult = await esClient.search({ index: 'orders-*', body: { query: { range: { created_at: { gte: 'now-24h' } } }, size: 0, track_total_hits: true } }); const esCount = esResult.hits.total.value; // Step 3: Compare expect(esCount).toBe(dbCount); }); it('revenue metric on dashboard matches transaction totals', async () => { const dbRevenue = await db.query( "SELECT SUM(total) as revenue FROM orders WHERE status = 'COMPLETED' AND created_at >= NOW() - INTERVAL '24 HOURS'" ); const expectedRevenue = parseFloat(dbRevenue.rows[0].revenue); const esResult = await esClient.search({ index: 'orders-*', body: { query: { bool: { must: [ { term: { status: 'COMPLETED' } }, { range: { created_at: { gte: 'now-24h' } } } ] } }, aggs: { total_revenue: { sum: { field: 'total' } } }, size: 0 } }); const dashboardRevenue = esResult.aggregations.total_revenue.value; // Allow small floating point tolerance expect(Math.abs(dashboardRevenue - expectedRevenue)).toBeLessThan(0.01); }); it('error rate percentage is calculated correctly', async () => { const esResult = await esClient.search({ index: 'logs-*', body: { query: { range: { '@timestamp': { gte: 'now-1h' } } }, aggs: { total: { value_count: { field: 'status_code' } }, errors: { filter: { range: { status_code: { gte: 500 } } }, aggs: { count: { value_count: { field: 'status_code' } } } } }, size: 0 } }); const total = esResult.aggregations.total.value; const errors = esResult.aggregations.errors.count.value; const expectedErrorRate = (errors / total) * 100; // Fetch what the dashboard shows via Kibana API const dashboardPanel = await kibanaApi.get('/api/saved_objects/visualization/error-rate-gauge'); const displayedErrorRate = await evaluateKibanaVisualization(dashboardPanel); expect(Math.abs(displayedErrorRate - expectedErrorRate)).toBeLessThan(0.1); }); }); ``` --- ## Elasticsearch Query Result Validation ```javascript describe('Elasticsearch Query Validation', () => { it('validates date histogram aggregation returns correct buckets', async () => { // Insert known test data const testDocs = []; for (let hour = 0; hour < 24; hour++) { const timestamp = new Date(); timestamp.setHours(hour, 0, 0, 0); testDocs.push({ '@timestamp': timestamp.toISOString(), service: 'order-api', status_code: hour % 5 === 0 ? 500 : 200, response_time: 100 + (hour * 10) }); } await esClient.bulk({ index: 'test-logs', body: testDocs.flatMap(doc => [{ index: {} }, doc]) }); await esClient.indices.refresh({ index: 'test-logs' }); // Run the same query the dashboard uses const result = await esClient.search({ index: 'test-logs', body: { query: { match_all: {} }, aggs: { requests_over_time: { date_histogram: { field: '@timestamp', fixed_interval: '1h' }, aggs: { avg_response: { avg: { field: 'response_time' } }, error_count: { filter: { range: { status_code: { gte: 500 } } } } } } }, size: 0 } }); const buckets = result.aggregations.requests_over_time.buckets; expect(buckets.length).toBe(24); // Verify specific bucket values const errorBuckets = buckets.filter(b => b.error_count.doc_count > 0); expect(errorBuckets.length).toBe(5); // Hours 0, 5, 10, 15, 20 }); it('validates term aggregation for top services', async () => { const result = await esClient.search({ index: 'logs-*', body: { query: { range: { '@timestamp': { gte: 'now-1h' } } }, aggs: { top_services: { terms: { field: 'service.keyword', size: 10 } } }, size: 0 } }); const services = result.aggregations.top_services.buckets; expect(services.length).toBeGreaterThan(0); // Each bucket should have reasonable doc counts for (const bucket of services) { expect(bucket.key).toBeDefined(); expect(bucket.doc_count).toBeGreaterThan(0); } }); }); ``` --- ## Kibana Dashboard Element Assertions ```javascript describe('Kibana Dashboard Visual Validation', () => { it('validates dashboard panels render without errors', async () => { await page.goto(`${kibanaUrl}/app/dashboards#/view/operations-overview`); // Wait for all panels to finish loading await page.waitForSelector('.embPanel__content', { state: 'visible' }); await page.waitForFunction(() => { const loaders = document.querySelectorAll('.euiLoadingSpinner'); return loaders.length === 0; }, { timeout: 30000 }); // Check no error icons on any panel const errorPanels = await page.locator('.embPanel--error').count(); expect(errorPanels).toBe(0); // Check no "No results found" where data is expected const noResultPanels = await page.locator('text="No results found"').count(); expect(noResultPanels).toBe(0); }); it('validates metric visualization shows correct value', async () => { await page.goto(`${kibanaUrl}/app/dashboards#/view/operations-overview`); await page.waitForLoadState('networkidle'); // Get the displayed metric value const metricValue = await page.locator('[data-test-subj="metricVis-total-orders"] .mtrVis__value').textContent(); const displayedCount = parseInt(metricValue.replace(/,/g, '')); // Compare with direct ES query const esResult = await esClient.count({ index: 'orders-*' }); expect(displayedCount).toBe(esResult.count); }); it('validates table visualization columns and sorting', async () => { await page.goto(`${kibanaUrl}/app/dashboards#/view/operations-overview`); await page.waitForLoadState('networkidle'); // Verify expected columns exist const headers = await page.locator('.euiTable th').allTextContents(); expect(headers).toContain('Service'); expect(headers).toContain('Error Rate'); expect(headers).toContain('P95 Latency'); // Verify sorting works await page.click('th:has-text("Error Rate")'); const firstRow = await page.locator('.euiTable tbody tr:first-child td').allTextContents(); const secondRow = await page.locator('.euiTable tbody tr:nth-child(2) td').allTextContents(); const firstErrorRate = parseFloat(firstRow[1]); const secondErrorRate = parseFloat(secondRow[1]); expect(firstErrorRate).toBeGreaterThanOrEqual(secondErrorRate); }); }); ``` --- ## Alert Rule Testing ```javascript describe('Alert Rule Validation', () => { it('fires alert when error rate exceeds threshold', async () => { // Generate errors to exceed the 5% threshold const requests = []; for (let i = 0; i < 100; i++) { requests.push({ '@timestamp': new Date().toISOString(), service: 'payment-api', status_code: i < 10 ? 500 : 200, // 10% error rate > 5% threshold response_time: 200 }); } await esClient.bulk({ index: 'logs-payment', body: requests.flatMap(doc => [{ index: {} }, doc]) }); await esClient.indices.refresh({ index: 'logs-payment' }); // Wait for alert evaluation cycle (typically 1 minute)