Skip to main content Home Creators proffesor-for-testing agentic-qe observability-testing-patterns
observability-testing-patterns Observability and monitoring validation patterns for dashboards, alerting, log aggregation, APM traces, and SLA/SLO verification. Use when testing monitoring infrastructure, dashboard accuracy, alert rules, or metric pipelines.
Jump to install Skills Marketplace Discover and explore AI skills built by the community.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
Show prompt details A direct command skips the review prompt. Inspect the source before running it.
npx skills add https://github.com/proffesor-for-testing/agentic-qe --skill observability-testing-patternsThe command stays on one line. Scroll horizontally to inspect it before copying.
Prefer a local copy? Download the files currently available to SkillsMP.
Download Zip Downloading... More from this repository Ruflo is a multi-agent orchestration platform for AI coding agents (Claude Code, Cursor, Codex, Copilot, Gemini, Amp, +12 more). Use this skill when the user wants to (1) install/init ruflo in a project, (2) run multi-agent swarms with hierarchical coordination, (3) use ruflo's 314+ MCP tools for memory, routing, hooks, sub-agents, or workflows, (4) check ruflo status/version/doctor health, or (5) discover which of ruflo's 30+ plugins fits their task.
Build dependency-aware execution plans for complex Agentic QE, Ruflo, integration, migration, or multi-stream engineering programs. Use when Codex must turn research or requirements into phased work, select a small AQE fleet, map critical paths and parallel streams, define acceptance gates, or sequence risky changes. Use aqe-plan-quality instead when the primary output is only a test or quality plan.
Conduct evidence-first technical research for Agentic QE, Ruflo, related ruvnet projects, or external tools. Use when Codex must investigate a repository, compare current upstream changes, trace dependencies and history, distinguish verified facts from inference, or synthesize findings into actionable engineering recommendations. Do not use for a simple known-answer lookup or an implementation-only task.
Related occupations SOC
Based on SOC occupation classification
name observability-testing-patterns description Observability and monitoring validation patterns for dashboards, alerting, log aggregation, APM traces, and SLA/SLO verification. Use when testing monitoring infrastructure, dashboard accuracy, alert rules, or metric pipelines. category specialized-testing priority high tokenEstimate 1600 agents ["qe-integration-tester","qe-performance-tester","qe-visual-tester"] implementation_status optimized optimization_version 1 last_optimized "2026-02-04T00:00:00.000Z" dependencies ["api-testing-patterns","shift-right-testing"] quick_reference_card true tags ["observability","monitoring","kibana","elasticsearch","dashboards","alerting","metrics","logging"] trust_tier 3 validation {"schema_path":"schemas/output.json","validator_path":"scripts/validate-config.json","eval_path":"evals/observability-testing-patterns.yaml"}
Observability Testing Patterns
Browser engine
Dashboard screenshot validation and alert-UI verification go through the qe-browser fleet skill (.claude/skills/qe-browser/). Vibium is installed by aqe init. Typical dashboard regression workflow:
vibium go "$GRAFANA_URL /d/api-latency"
vibium wait load
node .claude/skills/qe-browser/scripts/assert.js --checks '[
{"kind": "selector_visible", "selector": ".panel-title"},
{"kind": "no_console_errors"},
{"kind": "no_failed_requests"},
{"kind": "element_count", "selector": ".panel", "op": ">=", "count": 4}
]'
node .claude/skills/qe-browser/scripts/visual-diff.js --name "grafana-api-latency"
<default_to_action>
When testing observability infrastructure, dashboards, or monitoring:
VALIDATE data accuracy (source data matches what the dashboard displays)
TEST alert rules fire correctly at defined thresholds
VERIFY log aggregation completeness (no missing logs across services)
TRACE distributed requests end-to-end through APM
MEASURE dashboard performance (render time, query latency)
CONFIRM SLA/SLO compliance through synthetic monitoring
TEST metric pipeline integrity from collection to display
Quick Pattern Selection:
Dashboard shows wrong numbers -> Data accuracy validation
Alerts not firing -> Alert rule threshold testing
Missing logs in Kibana -> Log aggregation completeness
Slow dashboard -> Dashboard performance testing
Broken traces -> APM trace validation
SLA disputes -> SLO compliance validation
Critical Success Factors:
Observability is only as good as the data it shows
A dashboard that lies is worse than no dashboard
Alert fatigue kills response times; test thresholds carefully
</default_to_action>
Quick Reference Card
When to Use
Validating dashboard data accuracy (Kibana, Grafana, Datadog)
Testing alert rule thresholds and notification delivery
Verifying log aggregation completeness across microservices
Validating distributed tracing (APM) correctness
Measuring SLA/SLO compliance
Testing metric pipeline integrity (collection -> aggregation -> display)
Testing Levels Level Purpose Dependencies Speed Query Validation Elasticsearch/PromQL query accuracy Data source Fast Dashboard Accuracy Visual matches source data Full stack Medium Alert Threshold Trigger and notification testing Alerting stack Medium Pipeline Integrity End-to-end metric flow Full pipeline Slower Performance Dashboard render time, query latency Full stack Slower
Critical Test Scenarios Scenario Must Test Example Data Accuracy Dashboard = source truth Order count on dashboard = DB count Alert Firing Threshold triggers alert Error rate > 5% fires PagerDuty Alert Recovery Auto-resolve when recovered Error rate drops below 5% clears alert Log Completeness All services emit logs 10 microservices, all logs in Kibana Trace Integrity Full request path visible Auth -> API -> DB -> Cache spans SLO Compliance Error budget tracking 99.9% availability over 30 days Time Accuracy Timestamps aligned Log timestamp matches event time
Tools
Dashboards : Kibana, Grafana, Datadog, New RelicSearch : Elasticsearch, OpenSearch, LokiMetrics : Prometheus, InfluxDB, CloudWatchTracing : Jaeger, Zipkin, Datadog APM, OpenTelemetryAlerting : PagerDuty, OpsGenie, AlertmanagerSynthetic : Datadog Synthetics, Checkly, Playwright
Agent Coordination
qe-integration-tester: Validate data pipelines, query accuracy, log completenessqe-performance-tester: Dashboard render performance, query latencyqe-visual-tester: Dashboard visual regression, layout accuracy
Dashboard Data Accuracy Validation
Compare Source Data to Dashboard describe ('Dashboard Data Accuracy' , () => {
it ('order count on dashboard matches database' , async () => {
const dbResult = await db.query (
"SELECT COUNT(*) as count FROM orders WHERE created_at >= NOW() - INTERVAL '24 HOURS'"
);
const dbCount = parseInt (dbResult.rows [0 ].count );
const esResult = await esClient.search ({
index : 'orders-*' ,
body : {
query : {
range : { created_at : { gte : 'now-24h' } }
},
size : 0 ,
track_total_hits : true
}
});
const esCount = esResult.hits .total .value ;
expect (esCount).toBe (dbCount);
});
it ('revenue metric on dashboard matches transaction totals' , async () => {
const dbRevenue = await db.query (
"SELECT SUM(total) as revenue FROM orders WHERE status = 'COMPLETED' AND created_at >= NOW() - INTERVAL '24 HOURS'"
);
const expectedRevenue = parseFloat (dbRevenue.rows [0 ].revenue );
const esResult = await esClient.search ({
index : 'orders-*' ,
body : {
query : {
bool : {
must : [
{ term : { status : 'COMPLETED' } },
{ range : { created_at : { gte : 'now-24h' } } }
]
}
},
aggs : {
total_revenue : { sum : { field : 'total' } }
},
size : 0
}
});
const dashboardRevenue = esResult.aggregations .total_revenue .value ;
expect (Math .abs (dashboardRevenue - expectedRevenue)).toBeLessThan (0.01 );
});
it ('error rate percentage is calculated correctly' , async () => {
const esResult = await esClient.search ({
index : 'logs-*' ,
body : {
query : { range : { '@timestamp' : { gte : 'now-1h' } } },
aggs : {
total : { value_count : { field : 'status_code' } },
errors : {
filter : { range : { status_code : { gte : 500 } } },
aggs : { count : { value_count : { field : 'status_code' } } }
}
},
size : 0
}
});
const total = esResult.aggregations .total .value ;
const errors = esResult.aggregations .errors .count .value ;
const expectedErrorRate = (errors / total) * 100 ;
const dashboardPanel = await kibanaApi.get ('/api/saved_objects/visualization/error-rate-gauge' );
const displayedErrorRate = await evaluateKibanaVisualization (dashboardPanel);
expect (Math .abs (displayedErrorRate - expectedErrorRate)).toBeLessThan (0.1 );
});
});
Elasticsearch Query Result Validation describe ('Elasticsearch Query Validation' , () => {
it ('validates date histogram aggregation returns correct buckets' , async () => {
const testDocs = [];
for (let hour = 0 ; hour < 24 ; hour++) {
const timestamp = new Date ();
timestamp.setHours (hour, 0 , 0 , 0 );
testDocs.push ({
'@timestamp' : timestamp.toISOString (),
service : 'order-api' ,
status_code : hour % 5 === 0 ? 500 : 200 ,
response_time : 100 + (hour * 10 )
});
}
await esClient.bulk ({
index : 'test-logs' ,
body : testDocs.flatMap (doc =>index : {} }, doc])
});
await esClient.indices .refresh ({ index : 'test-logs' });
const result = await esClient.search ({
index : 'test-logs' ,
body : {
query : { match_all : {} },
aggs : {
requests_over_time : {
date_histogram : { field : '@timestamp' , fixed_interval : '1h' },
aggs : {
avg_response : { avg : { field : 'response_time' } },
error_count : {
filter : { range : { status_code : { gte : 500 } } }
}
}
}
},
size : 0
}
});
const buckets = result.aggregations .requests_over_time .buckets ;
expect (buckets.length ).toBe (24 );
const errorBuckets = buckets.filter (b =>error_count .doc_count > 0 );
expect (errorBuckets.length ).toBe (5 );
});
it ('validates term aggregation for top services' , async () => {
const result = await esClient.search ({
index : 'logs-*' ,
body : {
query : { range : { '@timestamp' : { gte : 'now-1h' } } },
aggs : {
top_services : {
terms : { field : 'service.keyword' , size : 10 }
}
},
size : 0
}
});
const services = result.aggregations .top_services .buckets ;
expect (services.length ).toBeGreaterThan (0 );
for (const bucket of services) {
expect (bucket.key ).toBeDefined ();
expect (bucket.doc_count ).toBeGreaterThan (0 );
}
});
});
Kibana Dashboard Element Assertions describe ('Kibana Dashboard Visual Validation' , () => {
it ('validates dashboard panels render without errors' , async () => {
await page.goto (`${kibanaUrl} /app/dashboards#/view/operations-overview` );
await page.waitForSelector ('.embPanel__content' , { state : 'visible' });
await page.waitForFunction (() => {
const loaders = document .querySelectorAll ('.euiLoadingSpinner' );
return loaders.length === 0 ;
}, { timeout : 30000 });
const errorPanels = await page.locator ('.embPanel--error' ).count ();
expect (errorPanels).toBe (0 );
const noResultPanels = await page.locator ('text="No results found"' ).count ();
expect (noResultPanels).toBe (0 );
});
it ('validates metric visualization shows correct value' , async () => {
await page.goto (`${kibanaUrl} /app/dashboards#/view/operations-overview` );
await page.waitForLoadState ('networkidle' );
const metricValue = await page.locator ('[data-test-subj="metricVis-total-orders"] .mtrVis__value' ).textContent ();
const displayedCount = parseInt (metricValue.replace (/,/g , '' ));
const esResult = await esClient.count ({ index : 'orders-*' });
expect (displayedCount).toBe (esResult.count );
});
it ('validates table visualization columns and sorting' , async () => {
await page.goto (`${kibanaUrl} /app/dashboards#/view/operations-overview` );
await page.waitForLoadState ('networkidle' );
const headers = await page.locator ('.euiTable th' ).allTextContents ();
expect (headers).toContain ('Service' );
expect (headers).toContain ('Error Rate' );
expect (headers).toContain ('P95 Latency' );
await page.click ('th:has-text("Error Rate")' );
const firstRow = await page.locator ('.euiTable tbody tr:first-child td' ).allTextContents ();
const secondRow = await page.locator ('.euiTable tbody tr:nth-child(2) td' ).allTextContents ();
const firstErrorRate = parseFloat (firstRow[1 ]);
const secondErrorRate = parseFloat (secondRow[1 ]);
expect (firstErrorRate).toBeGreaterThanOrEqual (secondErrorRate);
});
});
Alert Rule Testing describe ('Alert Rule Validation' , () => {
it ('fires alert when error rate exceeds threshold' , async () => {
const requests = [];
for (let i = 0 ; i < 100 ; i++) {
requests.push ({
'@timestamp' : new Date ().toISOString (),
service : 'payment-api' ,
status_code : i < 10 ? 500 : 200 ,
response_time : 200
});
}
await esClient.bulk ({
index : 'logs-payment' ,
body : requests.flatMap (doc =>index : {} }, doc])
});
await esClient.indices .refresh ({ index : 'logs-payment' });
await sleep (90000 );
const alerts = await alertManager.getActiveAlerts ({
filter : 'alertname="HighErrorRate" AND service="payment-api"'
});
expect (alerts.length ).toBeGreaterThan (0 );
expect (alerts[0 ].labels .severity ).toBe ('critical' );
});
it ('alert auto-resolves when condition clears' , async () => {
await injectErrors ('payment-api' , { count : 50 , total : 100 });
await sleep (90000 );
let alerts = await alertManager.getActiveAlerts ({ filter : 'alertname="HighErrorRate"' });
expect (alerts.length ).toBeGreaterThan (0 );
await injectSuccessRequests ('payment-api' , { count : 1000 });
await sleep (90000 );
alerts = await alertManager.getActiveAlerts ({ filter : 'alertname="HighErrorRate"' });
expect (alerts.length ).toBe (0 );
});
it ('alert notification reaches correct channel' , async () => {
const notifications = [];
const subscription = pagerDutyMock.onIncident ((incident ) => {
notifications.push (incident);
});
await injectErrors ('critical-service' , { count : 50 , total : 100 });
await sleep (120000 );
expect (notifications.length ).toBeGreaterThan (0 );
expect (notifications[0 ].service .name ).toBe ('critical-service' );
expect (notifications[0 ].urgency ).toBe ('high' );
subscription.unsubscribe ();
});
it ('alert does not fire for brief transient spikes' , async () => {
await injectErrors ('api-service' , { count : 20 , total : 50 , duration : 30000 });
await sleep (120000 );
const alerts = await alertManager.getActiveAlerts ({ filter : 'alertname="HighErrorRate"' });
expect (alerts.length ).toBe (0 );
});
});
Log Aggregation Completeness describe ('Log Aggregation Completeness' , () => {
it ('all microservice logs appear in centralized index' , async () => {
const traceId = uuid ();
const services = ['api-gateway' , 'auth-service' , 'order-service' , 'payment-service' , 'notification-service' ];
for (const service of services) {
await serviceLogEmitter.emit (service, {
level : 'INFO' ,
message : `Completeness test - ${traceId} ` ,
traceId,
timestamp : new Date ().toISOString ()
});
}
await sleep (15000 );
const result = await esClient.search ({
index : 'logs-*' ,
body : {
query : { term : { 'traceId.keyword' : traceId } },
size : 100
}
});
const foundServices = result.hits .hits .map (h =>_source .service );
for (const service of services) {
expect (foundServices).toContain (service);
}
expect (foundServices.length ).toBe (services.length );
});
it ('logs retain correct structure after pipeline processing' , async () => {
const testLog = {
level : 'ERROR' ,
message : 'Payment declined' ,
traceId : uuid (),
userId : 'user-123' ,
orderId : 'order-456' ,
errorCode : 'INSUFFICIENT_FUNDS' ,
timestamp : new Date ().toISOString ()
};
await serviceLogEmitter.emit ('payment-service' , testLog);
await sleep (10000 );
const result = await esClient.search ({
index : 'logs-*' ,
body : { query : { term : { 'traceId.keyword' : testLog.traceId } } }
});
expect (result.hits .hits .length ).toBe (1 );
const indexed = result.hits .hits [0 ]._source ;
expect (indexed.level ).toBe ('ERROR' );
expect (indexed.message ).toBe ('Payment declined' );
expect (indexed.userId ).toBe ('user-123' );
expect (indexed.orderId ).toBe ('order-456' );
expect (indexed.errorCode ).toBe ('INSUFFICIENT_FUNDS' );
});
it ('detects log volume drops indicating pipeline issues' , async () => {
const baseline = await esClient.count ({
index : 'logs-*' ,
body : { query : { range : { '@timestamp' : { gte : 'now-2h' , lt : 'now-1h' } } } }
});
const current = await esClient.count ({
index : 'logs-*' ,
body : { query : { range : { '@timestamp' : { gte : 'now-1h' } } } }
});
const ratio = current.count / baseline.count ;
expect (ratio).toBeGreaterThan (0.5 );
});
});
APM Trace Validation describe ('Distributed Trace Validation' , () => {
it ('captures complete trace across all services' , async () => {
const response = await httpClient.post ('/api/orders' , {
customerId : 'CUST-TRACE' ,
items : [{ sku : 'ITEM-1' , qty : 1 }]
});
const traceId = response.headers ['x-trace-id' ];
expect (traceId).toBeDefined ();
await sleep (10000 );
const trace = await jaegerClient.getTrace (traceId);
const spanNames = trace.spans .map (s =>operationName );
expect (spanNames).toContain ('POST /api/orders' );
expect (spanNames).toContain ('auth.validateToken' );
expect (spanNames).toContain ('order.create' );
expect (spanNames).toContain ('payment.authorize' );
expect (spanNames).toContain ('inventory.reserve' );
expect (spanNames).toContain ('db.insert orders' );
const apiSpan = trace.spans .find (s =>operationName === 'POST /api/orders' );
const authSpan = trace.spans .find (s =>operationName === 'auth.validateToken' );
expect (authSpan.references [0 ].refType ).toBe ('CHILD_OF' );
expect (authSpan.references [0 ].spanID ).toBe (apiSpan.spanID );
});
it ('traces capture error spans correctly' , async () => {
const response = await httpClient.post ('/api/orders' , {
customerId : 'INVALID-CUSTOMER' ,
items : [{ sku : 'ITEM-1' , qty : 1 }]
});
const traceId = response.headers ['x-trace-id' ];
await sleep (10000 );
const trace = await jaegerClient.getTrace (traceId);
const errorSpan = trace.spans .find (s =>tags .some (t =>key === 'error' && t.value === true ));
expect (errorSpan).toBeDefined ();
expect (errorSpan.logs ).toContainEqual (
expect.objectContaining ({
fields : expect.arrayContaining ([
expect.objectContaining ({ key : 'error.message' })
])
})
);
});
it ('validates trace sampling rate' , async () => {
const requestCount = 100 ;
const traceIds = [];
for (let i = 0 ; i < requestCount; i++) {
const resp = await httpClient.get ('/api/health' );
if (resp.headers ['x-trace-id' ]) {
traceIds.push (resp.headers ['x-trace-id' ]);
}
}
await sleep (15000 );
let tracesFound = 0 ;
for (const traceId of traceIds) {
try {
await jaegerClient.getTrace (traceId);
tracesFound++;
} catch (e) {
}
}
const samplingRate = tracesFound / requestCount;
expect (samplingRate).toBeGreaterThan (0.05 );
expect (samplingRate).toBeLessThan (0.20 );
});
});
SLA/SLO Validation describe ('SLA/SLO Compliance Validation' , () => {
it ('validates 99.9% availability SLO over 30 days' , async () => {
const result = await prometheusClient.query (
'avg_over_time(up{job="api-service"}[30d])'
);
const availability = parseFloat (result.data .result [0 ].value [1 ]) * 100 ;
expect (availability).toBeGreaterThanOrEqual (99.9 );
const totalMinutes = 30 * 24 * 60 ;
const allowedDowntime = totalMinutes * 0.001 ;
const actualDowntime = totalMinutes * (1 - availability / 100 );
const errorBudgetRemaining = ((allowedDowntime - actualDowntime) / allowedDowntime) * 100 ;
expect (errorBudgetRemaining).toBeGreaterThan (0 );
console .log (`Error budget remaining: ${errorBudgetRemaining.toFixed(1 )} %` );
});
it ('validates P95 latency SLO' , async () => {
const result = await prometheusClient.query (
'histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service="api-service"}[24h])) by (le))'
);
const p95Latency = parseFloat (result.data .result [0 ].value [1 ]) * 1000 ;
expect (p95Latency).toBeLessThan (500 );
});
it ('runs synthetic monitoring check for uptime' , async () => {
const endpoints = [
{ url : '/api/health' , expectedStatus : 200 , maxLatency : 200 },
{ url : '/api/orders' , expectedStatus : 401 , maxLatency : 300 },
{ url : '/api/products' , expectedStatus : 200 , maxLatency : 500 }
];
const results = [];
for (const endpoint of endpoints) {
const start = Date .now ();
const response = await httpClient.get (endpoint.url );
const latency = Date .now () - start;
results.push ({
url : endpoint.url ,
status : response.status ,
latency,
statusMatch : response.status === endpoint.expectedStatus ,
latencyOk : latency <= endpoint.maxLatency
});
}
for (const result of results) {
expect (result.statusMatch ).toBe (true );
expect (result.latencyOk ).toBe (true );
}
});
});
Metric Pipeline Integrity describe ('Metric Pipeline - Collection to Display' , () => {
it ('validates custom metric flows from app to Prometheus to Grafana' , async () => {
const metricName = 'test_orders_processed_total' ;
const expectedValue = 42 ;
await appMetrics.set (metricName, expectedValue, { service : 'test' });
await sleep (20000 );
const promResult = await prometheusClient.query (`${metricName} {service="test"}` );
const promValue = parseFloat (promResult.data .result [0 ].value [1 ]);
expect (promValue).toBe (expectedValue);
const grafanaResult = await grafanaApi.post ('/api/ds/query' , {
queries : [{
datasource : { type : 'prometheus' },
expr : `${metricName} {service="test"}` ,
refId : 'A'
}]
});
const grafanaValue = parseFloat (grafanaResult.body .results .A .frames [0 ].data .values [1 ][0 ]);
expect (grafanaValue).toBe (expectedValue);
});
it ('validates histogram metric percentile accuracy' , async () => {
const latencies = [10 , 20 , 30 , 50 , 100 , 200 , 300 , 500 , 1000 , 2000 ];
for (const latency of latencies) {
await appMetrics.observe ('http_request_duration_ms' , latency, { endpoint : '/test' });
}
await sleep (20000 );
const p50 = await prometheusClient.query (
'histogram_quantile(0.5, rate(http_request_duration_ms_bucket{endpoint="/test"}[5m]))'
);
const p99 = await prometheusClient.query (
'histogram_quantile(0.99, rate(http_request_duration_ms_bucket{endpoint="/test"}[5m]))'
);
const p50Value = parseFloat (p50.data .result [0 ].value [1 ]);
const p99Value = parseFloat (p99.data .result [0 ].value [1 ]);
expect (p50Value).toBeGreaterThan (50 );
expect (p50Value).toBeLessThan (300 );
expect (p99Value).toBeGreaterThan (1000 );
});
});
Dashboard Performance Testing describe ('Dashboard Performance' , () => {
it ('dashboard loads within acceptable time' , async () => {
const start = Date .now ();
await page.goto (`${kibanaUrl} /app/dashboards#/view/operations-overview` );
await page.waitForFunction (() => {
const spinners = document .querySelectorAll ('.euiLoadingSpinner' );
return spinners.length === 0 ;
}, { timeout : 30000 });
const loadTime = Date .now () - start;
expect (loadTime).toBeLessThan (10000 );
});
it ('dashboard handles large time range without timeout' , async () => {
await page.goto (`${kibanaUrl} /app/dashboards#/view/operations-overview?_g=(time:(from:now-30d,to:now))` );
await page.waitForFunction (() => {
const errors = document .querySelectorAll ('.embPanel--error' );
const spinners = document .querySelectorAll ('.euiLoadingSpinner' );
return errors.length === 0 && spinners.length === 0 ;
}, { timeout : 60000 });
const errorPanels = await page.locator ('.embPanel--error' ).count ();
expect (errorPanels).toBe (0 );
});
it ('Elasticsearch query performance is within bounds' , async () => {
const queries = [
{ name : 'date_histogram' , body : { aggs : { over_time : { date_histogram : { field : '@timestamp' , fixed_interval : '1h' } } }, size : 0 } },
{ name : 'terms_agg' , body : { aggs : { top_services : { terms : { field : 'service.keyword' , size : 20 } } }, size : 0 } },
{ name : 'percentiles' , body : { aggs : { latency : { percentiles : { field : 'response_time' , percents : [50 , 90 , 95 , 99 ] } } }, size : 0 } }
];
for (const query of queries) {
const start = Date .now ();
await esClient.search ({ index : 'logs-*' , body : { query : { range : { '@timestamp' : { gte : 'now-24h' } } }, ...query.body } });
const elapsed = Date .now () - start;
expect (elapsed).toBeLessThan (5000 );
}
});
});
Time-Series Data Accuracy describe ('Time-Series Data Accuracy' , () => {
it ('validates no data gaps in time-series metrics' , async () => {
const result = await prometheusClient.queryRange (
'up{job="api-service"}' ,
{ start : 'now-24h' , end : 'now' , step : '5m' }
);
const values = result.data .result [0 ].values ;
const expectedPoints = (24 * 60 ) / 5 ;
expect (values.length ).toBeGreaterThan (expectedPoints * 0.95 );
for (let i = 1 ; i < values.length ; i++) {
const gap = values[i][0 ] - values[i - 1 ][0 ];
expect (gap).toBeLessThanOrEqual (900 );
}
});
it ('validates timestamp alignment across sources' , async () => {
const eventTime = new Date ();
const traceId = uuid ();
await httpClient.post ('/api/test-event' , { traceId });
await sleep (15000 );
const logResult = await esClient.search ({
index : 'logs-*' ,
body : { query : { term : { 'traceId.keyword' : traceId } } }
});
const logTimestamp = new Date (logResult.hits .hits [0 ]._source ['@timestamp' ]);
const diff = Math .abs (logTimestamp.getTime () - eventTime.getTime ());
expect (diff).toBeLessThan (5000 );
});
});
Best Practices
Do This
Compare dashboard values against source-of-truth databases
Test alert thresholds with known data to verify exact firing conditions
Validate log completeness by injecting traceable test events
Test alert recovery (auto-resolve) not just alert firing
Monitor log volume as a proxy for pipeline health
Validate sampling rates for APM traces
Test dashboards at realistic time ranges (not just last 15 minutes)
Avoid This
Trusting dashboard numbers without source validation
Testing alerts only with manual threshold checks in the UI
Ignoring log pipeline latency (logs may take seconds to minutes to appear)
Skipping alert fatigue testing (too many false positives)
Assuming metrics are accurate without end-to-end pipeline validation
Testing only with small data volumes (performance issues appear at scale)
Forgetting to test alert notification delivery (PagerDuty, Slack, email)
Agent-Assisted Observability Testing
await Task ("Dashboard Data Accuracy Validation" , {
dashboard : 'operations-overview' ,
panels : ['order-count' , 'revenue-total' , 'error-rate' ],
sourceDatabase : 'orders-db' ,
compareFields : ['count' , 'sum(total)' , 'error_percentage' ],
tolerance : 0.01
}, "qe-integration-tester" );
await Task ("Dashboard Performance Benchmark" , {
dashboardUrl : 'http://kibana:5601/app/dashboards#/view/operations-overview' ,
timeRanges : ['15m' , '1h' , '24h' , '7d' , '30d' ],
maxLoadTime : 10000 ,
maxQueryTime : 5000 ,
captureScreenshots : true
}, "qe-performance-tester" );
await Task ("Dashboard Visual Regression" , {
dashboardUrl : 'http://kibana:5601/app/dashboards#/view/operations-overview' ,
baselineScreenshots : 'baseline/dashboards/' ,
threshold : 0.05 ,
ignoreRegions : ['timestamp-header' , 'dynamic-counters' ]
}, "qe-visual-tester" );
await Task ("Alert Rule Comprehensive Test" , {
alertRules : ['HighErrorRate' , 'HighLatency' , 'ServiceDown' ],
testFiring : true ,
testRecovery : true ,
testNotificationChannel : true ,
validateSilencing : true
}, "qe-integration-tester" );
Agent Coordination Hints
Memory Namespace aqe/observability-testing/
dashboards/ - Dashboard test results and screenshots
alerts/ - Alert rule test outcomes
logs/ - Log completeness validation results
traces/ - APM trace validation results
slo/ - SLA/SLO compliance metrics
pipelines/ - Metric pipeline integrity checks
performance/ - Dashboard and query performance benchmarks
Fleet Coordination const observabilityFleet = await FleetManager .coordinate ({
strategy : 'observability-testing' ,
agents : [
'qe-integration-tester' ,
'qe-performance-tester' ,
'qe-visual-tester'
],
topology : 'mesh'
});
await observabilityFleet.execute ({
targets : [
{ type : 'dashboard' , id : 'operations-overview' , checks : ['accuracy' , 'performance' , 'visual' ] },
{ type : 'alerts' , rules : ['HighErrorRate' , 'HighLatency' ], checks : ['fire' , 'resolve' , 'notify' ] },
{ type : 'logs' , services : ['api' , 'auth' , 'payment' ], checks : ['completeness' , 'structure' ] },
{ type : 'traces' , endpoints : ['/api/orders' ], checks : ['spans' , 'errors' , 'sampling' ] }
]
});
Related Skills
Remember Observability testing is about proving that your monitoring tells the truth. A dashboard that shows green when the system is on fire is worse than no dashboard at all. Validate data accuracy by comparing against source databases, test alert thresholds with controlled data injection, and verify log completeness by tracing known events through the entire pipeline.
With Agents: Agents automate the tedious comparison of dashboard values against source databases, systematically test alert thresholds with synthetic load, and validate log pipeline completeness across all services. Use agents to continuously verify that your observability stack is trustworthy.
