qe-observability-testing-patterns

inclusion	auto
name	qe-observability-testing-patterns
description	Observability and monitoring validation patterns for dashboards, alerting, log aggregation, APM traces, and SLA/SLO verification. Use when testing monitoring infrastructure, dashboard accuracy, alert rules, or metric pipelines.
tags	["observability","monitoring","kibana","elasticsearch","dashboards","alerting","metrics","logging"]

Observability Testing Patterns

<default_to_action> When testing observability infrastructure, dashboards, or monitoring:

1. VALIDATE data accuracy (source data matches what the dashboard displays)
2. TEST alert rules fire correctly at defined thresholds
3. VERIFY log aggregation completeness (no missing logs across services)
4. TRACE distributed requests end-to-end through APM
5. MEASURE dashboard performance (render time, query latency)
6. CONFIRM SLA/SLO compliance through synthetic monitoring
7. TEST metric pipeline integrity from collection to display

Quick Pattern Selection:

• Dashboard shows wrong numbers -> Data accuracy validation
• Alerts not firing -> Alert rule threshold testing
• Missing logs in Kibana -> Log aggregation completeness
• Slow dashboard -> Dashboard performance testing
• Broken traces -> APM trace validation
• SLA disputes -> SLO compliance validation

Critical Success Factors:

• Observability is only as good as the data it shows
• A dashboard that lies is worse than no dashboard
• Alert fatigue kills response times; test thresholds carefully </default_to_action>

Quick Reference Card

When to Use

• Validating dashboard data accuracy (Kibana, Grafana, Datadog)
• Testing alert rule thresholds and notification delivery
• Verifying log aggregation completeness across microservices
• Validating distributed tracing (APM) correctness
• Measuring SLA/SLO compliance
• Testing metric pipeline integrity (collection -> aggregation -> display)

Testing Levels

Level	Purpose	Dependencies	Speed
Query Validation	Elasticsearch/PromQL query accuracy	Data source	Fast
Dashboard Accuracy	Visual matches source data	Full stack	Medium
Alert Threshold	Trigger and notification testing	Alerting stack	Medium
Pipeline Integrity	End-to-end metric flow	Full pipeline	Slower
Performance	Dashboard render time, query latency	Full stack	Slower

Critical Test Scenarios

Scenario	Must Test	Example
Data Accuracy	Dashboard = source truth	Order count on dashboard = DB count
Alert Firing	Threshold triggers alert	Error rate > 5% fires PagerDuty
Alert Recovery	Auto-resolve when recovered	Error rate drops below 5% clears alert
Log Completeness	All services emit logs	10 microservices, all logs in Kibana
Trace Integrity	Full request path visible	Auth -> API -> DB -> Cache spans
SLO Compliance	Error budget tracking	99.9% availability over 30 days
Time Accuracy	Timestamps aligned	Log timestamp matches event time

Tools

• Dashboards: Kibana, Grafana, Datadog, New Relic
• Search: Elasticsearch, OpenSearch, Loki
• Metrics: Prometheus, InfluxDB, CloudWatch
• Tracing: Jaeger, Zipkin, Datadog APM, OpenTelemetry
• Alerting: PagerDuty, OpsGenie, Alertmanager
• Synthetic: Datadog Synthetics, Checkly, Playwright

Agent Coordination

• qe-integration-tester: Validate data pipelines, query accuracy, log completeness
• qe-performance-tester: Dashboard render performance, query latency
• qe-visual-tester: Dashboard visual regression, layout accuracy

---

Dashboard Data Accuracy Validation

Compare Source Data to Dashboard

describe('Dashboard Data Accuracy', () => {
  it('order count on dashboard matches database', async () => {
    // Step 1: Get ground truth from source database
    const dbResult = await db.query(
      "SELECT COUNT(*) as count FROM orders WHERE created_at >= NOW() - INTERVAL '24 HOURS'"
    );
    const dbCount = parseInt(dbResult.rows[0].count);

    // Step 2: Query Elasticsearch (same data source as dashboard)
    const esResult = await esClient.search({
      index: 'orders-*',
      body: {
        query: {
          range: { created_at: { gte: 'now-24h' } }
        },
        size: 0,
        track_total_hits: true
      }
    });
    const esCount = esResult.hits.total.value;

    // Step 3: Compare
    expect(esCount).toBe(dbCount);
  });

  it('revenue metric on dashboard matches transaction totals', async () => {
    const dbRevenue = await db.query(
      "SELECT SUM(total) as revenue FROM orders WHERE status = 'COMPLETED' AND created_at >= NOW() - INTERVAL '24 HOURS'"
    );
     expectedRevenue = (dbRevenue.[].);

     esResult =  esClient.({
      : ,
      : {
        : {
          : {
            : [
              { : { :  } },
              { : { : { :  } } }
            ]
          }
        },
        : {
          : { : { :  } }
        },
        : 
      }
    });
     dashboardRevenue = esResult...;

    
    (.(dashboardRevenue - expectedRevenue)).();
  });

  (,  () => {
     esResult =  esClient.({
      : ,
      : {
        : { : { : { :  } } },
        : {
          : { : { :  } },
          : {
            : { : { : { :  } } },
            : { : { : { :  } } }
          }
        },
        : 
      }
    });

     total = esResult...;
     errors = esResult....;
     expectedErrorRate = (errors / total) * ;

    
     dashboardPanel =  kibanaApi.();
     displayedErrorRate =  (dashboardPanel);

    (.(displayedErrorRate - expectedErrorRate)).();
  });
});

---

Elasticsearch Query Result Validation

describe('Elasticsearch Query Validation', () => {
  it('validates date histogram aggregation returns correct buckets', async () => {
    // Insert known test data
    const testDocs = [];
    for (let hour = 0; hour < 24; hour++) {
      const timestamp = new Date();
      timestamp.setHours(hour, 0, 0, 0);
      testDocs.push({
        '@timestamp': timestamp.toISOString(),
        service: 'order-api',
        status_code: hour % 5 === 0 ? 500 : 200,
        response_time: 100 + (hour * 10)
      });
    }

    await esClient.bulk({
      index: 'test-logs',
      body: testDocs.flatMap(doc => [{ index: {} }, doc])
    });
    await esClient.indices.refresh({ index: 'test-logs' });

    // Run the same query the dashboard uses
    const result = await esClient.({
      : ,
      : {
        : { : {} },
        : {
          : {
            : { : , :  },
            : {
              : { : { :  } },
              : {
                : { : { : { :  } } }
              }
            }
          }
        },
        : 
      }
    });

     buckets = result...;
    (buckets.).();

    
     errorBuckets = buckets.( b.. > );
    (errorBuckets.).(); 
  });

  (,  () => {
     result =  esClient.({
      : ,
      : {
        : { : { : { :  } } },
        : {
          : {
            : { : , :  }
          }
        },
        : 
      }
    });

     services = result...;
    (services.).();

    
     ( bucket  services) {
      (bucket.).();
      (bucket.).();
    }
  });
});

---

Kibana Dashboard Element Assertions

describe('Kibana Dashboard Visual Validation', () => {
  it('validates dashboard panels render without errors', async () => {
    await page.goto(`${kibanaUrl}/app/dashboards#/view/operations-overview`);

    // Wait for all panels to finish loading
    await page.waitForSelector('.embPanel__content', { state: 'visible' });
    await page.waitForFunction(() => {
      const loaders = document.querySelectorAll('.euiLoadingSpinner');
      return loaders.length === 0;
    }, { timeout: 30000 });

    // Check no error icons on any panel
    const errorPanels = await page.locator('.embPanel--error').count();
    expect(errorPanels).toBe(0);

    // Check no "No results found" where data is expected
    const noResultPanels = await page.locator('text="No results found"').count();
    expect(noResultPanels).toBe(0);
  });

  (,  () => {
     page.();
     page.();

    
     metricValue =  page.().();
     displayedCount = (metricValue.(, ));

    
     esResult =  esClient.({ :  });

    (displayedCount).(esResult.);
  });

  (,  () => {
     page.();
     page.();

    
     headers =  page.().();
    (headers).();
    (headers).();
    (headers).();

    
     page.();
     firstRow =  page.().();
     secondRow =  page.().();

     firstErrorRate = (firstRow[]);
     secondErrorRate = (secondRow[]);
    (firstErrorRate).(secondErrorRate);
  });
});

---

Alert Rule Testing

describe('Alert Rule Validation', () => {
  it('fires alert when error rate exceeds threshold', async () => {
    // Generate errors to exceed the 5% threshold
    const requests = [];
    for (let i = 0; i < 100; i++) {
      requests.push({
        '@timestamp': new Date().toISOString(),
        service: 'payment-api',
        status_code: i < 10 ? 500 : 200, // 10% error rate > 5% threshold
        response_time: 200
      });
    }

    await esClient.bulk({
      index: 'logs-payment',
      body: requests.flatMap(doc => [{ index: {} }, doc])
    });
    await esClient.indices.refresh({ index: 'logs-payment' });

    // Wait for alert evaluation cycle (typically 1 minute)
    await sleep(90000);

    // Check alert was fired
    const alerts = await alertManager.getActiveAlerts({
      : 
    });
    (alerts.).();
    (alerts[]..).();
  });

  (,  () => {
    
     (, { : , :  });
     ();

     alerts =  alertManager.({ :  });
    (alerts.).();

    
     (, { :  });
     ();

    
    alerts =  alertManager.({ :  });
    (alerts.).();
  });

  (,  () => {
    
     notifications = [];
     subscription = pagerDutyMock.( {
      notifications.(incident);
    });

    
     (, { : , :  });
     ();

    (notifications.).();
    (notifications[]..).();
    (notifications[].).();

    subscription.();
  });

  (,  () => {
    
     (, { : , : , :  });
     ();

     alerts =  alertManager.({ :  });
    (alerts.).(); 
  });
});

---

Log Aggregation Completeness

describe('Log Aggregation Completeness', () => {
  it('all microservice logs appear in centralized index', async () => {
    const traceId = uuid();
    const services = ['api-gateway', 'auth-service', 'order-service', 'payment-service', 'notification-service'];

    // Generate a log entry with known traceId in each service
    for (const service of services) {
      await serviceLogEmitter.emit(service, {
        level: 'INFO',
        message: `Completeness test - ${traceId}`,
        traceId,
        timestamp: new Date().toISOString()
      });
    }

    // Wait for log pipeline to process (Filebeat -> Logstash -> Elasticsearch)
    await sleep(15000);

    // Query Elasticsearch for the trace ID
    const result = await esClient.search({
      index: 'logs-*',
      body: {
        query: { term: { 'traceId.keyword': traceId } },
        size: 100
      }
    });

    const foundServices = result...( h..);

    
     ( service  services) {
      (foundServices).(service);
    }
    (foundServices.).(services.);
  });

  (,  () => {
     testLog = {
      : ,
      : ,
      : (),
      : ,
      : ,
      : ,
      :  ().()
    };

     serviceLogEmitter.(, testLog);
     ();

     result =  esClient.({
      : ,
      : { : { : { : testLog. } } }
    });

    (result...).();
     indexed = result..[].;

    
    (indexed.).();
    (indexed.).();
    (indexed.).();
    (indexed.).();
    (indexed.).();
  });

  (,  () => {
    
     baseline =  esClient.({
      : ,
      : { : { : { : { : , :  } } } }
    });

     current =  esClient.({
      : ,
      : { : { : { : { :  } } } }
    });

    
     ratio = current. / baseline.;
    (ratio).();
  });
});

---

APM Trace Validation

describe('Distributed Trace Validation', () => {
  it('captures complete trace across all services', async () => {
    // Make a request that traverses multiple services
    const response = await httpClient.post('/api/orders', {
      customerId: 'CUST-TRACE',
      items: [{ sku: 'ITEM-1', qty: 1 }]
    });

    const traceId = response.headers['x-trace-id'];
    expect(traceId).toBeDefined();

    // Wait for trace to be indexed
    await sleep(10000);

    // Query Jaeger/APM for the trace
    const trace = await jaegerClient.getTrace(traceId);

    // Verify all expected spans exist
    const spanNames = trace.spans.map(s => s.operationName);
    expect(spanNames).toContain('POST /api/orders');
    expect(spanNames).toContain('auth.validateToken');
    expect(spanNames).toContain('order.create');
    (spanNames).();
    (spanNames).();
    (spanNames).();

    
     apiSpan = trace..( s. === );
     authSpan = trace..( s. === );
    (authSpan.[].).();
    (authSpan.[].).(apiSpan.);
  });

  (,  () => {
    
     response =  httpClient.(, {
      : ,
      : [{ : , :  }]
    });

     traceId = response.[];
     ();

     trace =  jaegerClient.(traceId);

    
     errorSpan = trace..( s..( t. ===  && t. === ));
    (errorSpan).();
    (errorSpan.).(
      expect.({
        : expect.([
          expect.({ :  })
        ])
      })
    );
  });

  (,  () => {
     requestCount = ;
     traceIds = [];

     ( i = ; i < requestCount; i++) {
       resp =  httpClient.();
       (resp.[]) {
        traceIds.(resp.[]);
      }
    }

     ();

     tracesFound = ;
     ( traceId  traceIds) {
       {
         jaegerClient.(traceId);
        tracesFound++;
      }  (e) {
        
      }
    }

    
     samplingRate = tracesFound / requestCount;
    (samplingRate).();
    (samplingRate).();
  });
});

---

SLA/SLO Validation

describe('SLA/SLO Compliance Validation', () => {
  it('validates 99.9% availability SLO over 30 days', async () => {
    const result = await prometheusClient.query(
      'avg_over_time(up{job="api-service"}[30d])'
    );
    const availability = parseFloat(result.data.result[0].value[1]) * 100;

    expect(availability).toBeGreaterThanOrEqual(99.9);

    // Calculate error budget remaining
    const totalMinutes = 30 * 24 * 60;
    const allowedDowntime = totalMinutes * 0.001; // 43.2 minutes
    const actualDowntime = totalMinutes * (1 - availability / 100);
    const errorBudgetRemaining = ((allowedDowntime - actualDowntime) / allowedDowntime) * 100;

    expect(errorBudgetRemaining).toBeGreaterThan(0);
    console.log(`Error budget remaining: ${errorBudgetRemaining.toFixed(1)}%`);
  });

  it('validates P95 latency SLO', async () => {
    const result =  prometheusClient.(
      
    );
     p95Latency = (result..[].[]) * ; 

    (p95Latency).(); 
  });

  (,  () => {
     endpoints = [
      { : , : , :  },
      { : , : , :  }, 
      { : , : , :  }
    ];

     results = [];
     ( endpoint  endpoints) {
       start = .();
       response =  httpClient.(endpoint.);
       latency = .() - start;

      results.({
        : endpoint.,
        : response.,
        latency,
        : response. === endpoint.,
        : latency <= endpoint.
      });
    }

    
     ( result  results) {
      (result.).();
      (result.).();
    }
  });
});

---

Metric Pipeline Integrity

describe('Metric Pipeline - Collection to Display', () => {
  it('validates custom metric flows from app to Prometheus to Grafana', async () => {
    // Step 1: Emit a known custom metric from application
    const metricName = 'test_orders_processed_total';
    const expectedValue = 42;
    await appMetrics.set(metricName, expectedValue, { service: 'test' });

    // Step 2: Wait for scrape interval (15s default)
    await sleep(20000);

    // Step 3: Query Prometheus directly
    const promResult = await prometheusClient.query(`${metricName}{service="test"}`);
    const promValue = parseFloat(promResult.data.result[0].value[1]);
    expect(promValue).toBe(expectedValue);

    // Step 4: Query Grafana datasource API (same as dashboard would)
    const grafanaResult = await grafanaApi.post('/api/ds/query', {
      queries: [{
        datasource: { type: 'prometheus' },
        expr: `${metricName}{service="test"}`,
        : 
      }]
    });

     grafanaValue = (grafanaResult....[]..[][]);
    (grafanaValue).(expectedValue);
  });

  (,  () => {
    
     latencies = [, , , , , , , , , ]; 
     ( latency  latencies) {
       appMetrics.(, latency, { :  });
    }

     ();

    
     p50 =  prometheusClient.(
      
    );
     p99 =  prometheusClient.(
      
    );

     p50Value = (p50..[].[]);
     p99Value = (p99..[].[]);

    
    (p50Value).();
    (p50Value).();

    
    (p99Value).();
  });
});

---

Dashboard Performance Testing

describe('Dashboard Performance', () => {
  it('dashboard loads within acceptable time', async () => {
    const start = Date.now();

    await page.goto(`${kibanaUrl}/app/dashboards#/view/operations-overview`);

    // Wait for all panels to finish loading
    await page.waitForFunction(() => {
      const spinners = document.querySelectorAll('.euiLoadingSpinner');
      return spinners.length === 0;
    }, { timeout: 30000 });

    const loadTime = Date.now() - start;

    expect(loadTime).toBeLessThan(10000); // Dashboard should load in under 10s
  });

  it('dashboard handles large time range without timeout', async () => {
    // Set time range to 30 days
    await page.goto(`${kibanaUrl}/app/dashboards#/view/operations-overview?_g=(time:(from:now-30d,to:now))`);

    // Should complete without timeout error
    await page.waitForFunction(() => {
       errors = .();
       spinners = .();
       errors. ===  && spinners. === ;
    }, { :  });

     errorPanels =  page.().();
    (errorPanels).();
  });

  (,  () => {
     queries = [
      { : , : { : { : { : { : , :  } } }, :  } },
      { : , : { : { : { : { : , :  } } }, :  } },
      { : , : { : { : { : { : , : [, , , ] } } }, :  } }
    ];

     ( query  queries) {
       start = .();
       esClient.({ : , : { : { : { : { :  } } }, ...query. } });
       elapsed = .() - start;

      (elapsed).(); 
    }
  });
});

---

Time-Series Data Accuracy

describe('Time-Series Data Accuracy', () => {
  it('validates no data gaps in time-series metrics', async () => {
    const result = await prometheusClient.queryRange(
      'up{job="api-service"}',
      { start: 'now-24h', end: 'now', step: '5m' }
    );

    const values = result.data.result[0].values;
    const expectedPoints = (24 * 60) / 5; // 288 data points for 24h at 5m intervals

    // Allow up to 5% missing data points
    expect(values.length).toBeGreaterThan(expectedPoints * 0.95);

    // Check for gaps longer than 15 minutes (3 consecutive missing points)
    for (let i = 1; i < values.length; i++) {
      const gap = values[i][0] - values[i - 1][0]; // timestamp difference
      expect(gap).toBeLessThanOrEqual(900); // No gap > 15 minutes
    }
  });

  it('validates timestamp alignment across sources', async () => {
    
     eventTime =  ();
     traceId = ();

     httpClient.(, { traceId });
     ();

    
     logResult =  esClient.({
      : ,
      : { : { : { : traceId } } }
    });
     logTimestamp =  (logResult..[].[]);

    
    
     diff = .(logTimestamp.() - eventTime.());
    (diff).();
  });
});

---

Best Practices

Do This

• Compare dashboard values against source-of-truth databases
• Test alert thresholds with known data to verify exact firing conditions
• Validate log completeness by injecting traceable test events
• Test alert recovery (auto-resolve) not just alert firing
• Monitor log volume as a proxy for pipeline health
• Validate sampling rates for APM traces
• Test dashboards at realistic time ranges (not just last 15 minutes)

Avoid This

• Trusting dashboard numbers without source validation
• Testing alerts only with manual threshold checks in the UI
• Ignoring log pipeline latency (logs may take seconds to minutes to appear)
• Skipping alert fatigue testing (too many false positives)
• Assuming metrics are accurate without end-to-end pipeline validation
• Testing only with small data volumes (performance issues appear at scale)
• Forgetting to test alert notification delivery (PagerDuty, Slack, email)

---

Agent-Assisted Observability Testing

// Data accuracy validation
await Task("Dashboard Data Accuracy Validation", {
  dashboard: 'operations-overview',
  panels: ['order-count', 'revenue-total', 'error-rate'],
  sourceDatabase: 'orders-db',
  compareFields: ['count', 'sum(total)', 'error_percentage'],
  tolerance: 0.01
}, "qe-integration-tester");

// Dashboard performance testing
await Task("Dashboard Performance Benchmark", {
  dashboardUrl: 'http://kibana:5601/app/dashboards#/view/operations-overview',
  timeRanges: ['15m', '1h', '24h', '7d', '30d'],
  maxLoadTime: 10000,
  maxQueryTime: 5000,
  captureScreenshots: true
}, "qe-performance-tester");

// Dashboard visual regression
await Task("Dashboard Visual Regression", {
  dashboardUrl: 'http://kibana:5601/app/dashboards#/view/operations-overview',
  baselineScreenshots: 'baseline/dashboards/',
  threshold: 0.05,
  ignoreRegions: [, ]
}, );


 (, {
  : [, , ],
  : ,
  : ,
  : ,
  : 
}, );

---

Agent Coordination Hints

Memory Namespace

aqe/observability-testing/
  dashboards/          - Dashboard test results and screenshots
  alerts/              - Alert rule test outcomes
  logs/                - Log completeness validation results
  traces/              - APM trace validation results
  slo/                 - SLA/SLO compliance metrics
  pipelines/           - Metric pipeline integrity checks
  performance/         - Dashboard and query performance benchmarks

Fleet Coordination

const observabilityFleet = await FleetManager.coordinate({
  strategy: 'observability-testing',
  agents: [
    'qe-integration-tester',  // Data accuracy, log completeness, alert rules
    'qe-performance-tester',  // Dashboard load time, query latency
    'qe-visual-tester'        // Dashboard visual regression
  ],
  topology: 'mesh'
});

await observabilityFleet.execute({
  targets: [
    { type: 'dashboard', id: 'operations-overview', checks: ['accuracy', 'performance', 'visual'] },
    { type: 'alerts', rules: ['HighErrorRate', 'HighLatency'], checks: ['fire', 'resolve', 'notify'] },
    { type: 'logs', services: ['api', 'auth', 'payment'], checks: ['completeness', 'structure'] },
    { type: 'traces', endpoints: ['/api/orders'], checks: ['spans', 'errors', ] }
  ]
});

---

Related Skills

• api-testing-patterns - API testing for services feeding observability
• shift-right-testing - Production monitoring and observability
• performance-testing - Performance baselines and SLO definition
• chaos-engineering-resilience - Fault injection to validate alerting
• visual-testing-advanced - Dashboard visual regression

---

Remember

Observability testing is about proving that your monitoring tells the truth. A dashboard that shows green when the system is on fire is worse than no dashboard at all. Validate data accuracy by comparing against source databases, test alert thresholds with controlled data injection, and verify log completeness by tracing known events through the entire pipeline.

With Agents: Agents automate the tedious comparison of dashboard values against source databases, systematically test alert thresholds with synthetic load, and validate log pipeline completeness across all services. Use agents to continuously verify that your observability stack is trustworthy.