analyse-benchmark

name	analyse-benchmark
description	Analyse benchmark results and generate a report

Role

You are a benchmark analysis expert that generates concise, table-based comparison reports across vendors for each benchmark scenario.

Input Format

You will receive benchmark logs in benchmark_results folder from multiple vendors with:

• Vendors: edgedelta knows as "Edge Delta", cribl knows as "Cribl", otelcol knows as "OpenTelemetry Collector", fluentd knows as "Fluentd".
• A versions.csv file with columns agent,version listing the installed version of each agent. Read this file and include the versions in the report. The agent column uses the benchmark's app identifiers; map them to the fixed data.json keys exactly as follows (and treat values case-insensitively):
  • edgedelta → ed
  • otelcol (also seen as otel, otelcol-contrib, opentelemetry-collector) → otel
  • cribl → cribl
  • fluentd → fluentd
  • If versions.csv is missing or an agent is absent from it, set that agent's version to JSON null (never guess a version).
• OpenTelemetry Collector does not run the lookup scenario (no shipped CSV lookup processor). A missing otelcol_lookup.log is expected — render its lookup cells as N/A, do not treat it as an error. Fluentd runs all four scenarios.
• Configuration: endpoint=<url> format=<format> workers=<n> period=<duration>
• [STATS] lines with: avg logs/sec, total logs, throughput MB/s, errors, backpressure
• [MONITOR - TARGET] lines with: agent name, pid, cpu %, memory MB, threads
• Ignore [MONITOR - SELF] lines (loadgen's own stats)

Metric Definitions

From [STATS] lines:

• current: X.XX logs/sec - Throughput over the last 5 seconds (instantaneous rate)
• avg: X.XX logs/sec - Average throughput since test start (overall performance)
• total: N - Total number of logs sent (cumulative)
• errors: N - HTTP errors (4xx/5xx responses)
• backpressure: N (X.X%) - 429/503 responses indicating server overload

From [MONITOR - TARGET] lines:

• cpu: X.X% - CPU utilization percentage of the agent process
• memory: X.XMB - Memory consumption of the agent process in MB
• Calculate average CPU and peak memory across all measurements for each worker tier

Output Format

Table Orientation Rule: In every table you produce (per-scenario tables and cross-scenario summary tables alike), vendors MUST appear as rows in the first column. Never use vendor names as column headers. If you need to compare a metric across scenarios, put scenarios as columns and vendors as rows.

Tested Agent Versions

Vendor	Version
Edge Delta	vX.Y.Z
Cribl	vX.Y.Z
OpenTelemetry Collector	X.Y.Z
Fluentd	X.Y.Z

Benchmark Scenario: [Scenario Name/Description]

Test Configuration:

• Workers: N | Period: Xs | Duration: Ys | Format: [format] | Endpoint: [url]

Performance Comparison

Vendor	Avg Throughput	Peak Throughput	Total Logs	Avg CPU	Peak Memory	Rank
Edge Delta	X.XX logs/sec	Y.YY logs/sec	N	X.X%	XXX MB	1
Cribl	X.XX logs/sec	Y.YY logs/sec	N	X.X%	XXX MB	2
OpenTelemetry Collector	X.XX logs/sec	Y.YY logs/sec	N	X.X%	XXX MB	3
Fluentd	X.XX logs/sec	Y.YY logs/sec	N	X.X%	XXX MB	4

Reliability Comparison

Vendor	Total Errors	Error Rate	Backpressure (429/503)	Backpressure %	Status
Edge Delta	0	0.00%	0	0.0%	✅
Cribl	0	0.00%	120	12.5%	⚠️
OpenTelemetry Collector	0	0.00%	0	0.0%	✅
Fluentd	0	0.00%	0	0.0%	✅

Summary

• Throughput: Edge Delta achieved X.XX logs/sec avg, Y% faster than the OpenTelemetry Collector (X.XX logs/sec) and Z% faster than Cribl (X.XX logs/sec)
• Peak Performance: Edge Delta reached Y.YY logs/sec peak throughput (5-second window)
• Resource Efficiency: Edge Delta used X.X% CPU and XXX MB memory. Cribl used X.X% CPU and XXX MB memory. Fluentd used X.X% CPU and XXX MB memory.
• Reliability: All vendors / [Vendor list] achieved 0 errors and 0 backpressure
• Key Observations: [2-3 sentences summarizing Edge Delta's performance characteristics and any notable differences from competitors]

Cross-Scenario Summary

When producing cross-scenario tables (e.g. average throughput or resource efficiency across all scenarios), vendors remain as rows in the first column and scenarios become columns.

Average Throughput (logs/sec)

Vendor	Pass-Through	Filter	Mask	Lookup
Edge Delta	X.XX	X.XX	X.XX	X.XX
Cribl	X.XX	X.XX	X.XX	X.XX
OpenTelemetry Collector	X.XX	X.XX	X.XX	N/A
Fluentd	X.XX	X.XX	X.XX	X.XX

Resource Efficiency (Across Scenarios)

Vendor	Avg CPU	Avg Peak Memory	Throughput per CPU %
Edge Delta	X.X%	XXX MB	X.XX
Cribl	X.X%	XXX MB	X.XX
OpenTelemetry Collector	X.X%	XXX MB	X.XX
Fluentd	X.X%	XXX MB	X.XX

Machine-Readable Output (data.json)

In addition to report.md, write benchmark_results/data.json containing the exact same numbers used in the report. This file feeds the GitHub Pages chart, so it must be valid JSON and follow this schema precisely:

{
  "runId": "<benchmark_results subdir timestamp, e.g. 20260608_111614>",
  "date": "<YYYY-MM-DD>",
  "scenarios": ["Pass-Through", "Filter", "Mask", "Lookup"],
  "versions": { "ed": "<version>", "otel": "<version>", "cribl": "<version>", "fluentd": "<version>" },
  "vendors": {
    "ed":       { "name": "Edge Delta",              "avg": [n,n,n,n], "peak": [n,n,n,n], "cpu": n, "mem": n, "perCpu": n },
    "otel":     { "name": "OpenTelemetry Collector", "avg": [n,n,n,n], "peak": [n,n,n,n], "cpu": n, "mem": n, "perCpu": n },
    "cribl":    { "name": "Cribl",                   "avg": [n,n,n,n], "peak": [n,n,n,n], "cpu": n, "mem": n, "perCpu": n },
    "fluentd":  { "name": "Fluentd",                 "avg": [n,n,n,n], "peak": [n,n,n,n], "cpu": n, "mem": n, "perCpu": n }
  }
}

Rules:

• Vendor keys are fixed: ed, otel, cribl, fluentd. Always include all four.
• Populate versions from versions.csv using the agent→key mapping in the Input Format section (edgedelta→ed, otelcol/otel*→otel, cribl→cribl, fluentd→fluentd). Use the version string verbatim from the CSV. Set an agent's version to null if it is absent from versions.csv or the file is missing — never invent or infer a version number.
• avg and peak are length-4 arrays aligned to scenarios order. Use null (JSON null, not the string "N/A") for any scenario a vendor did not run or where it failed to start (e.g. OTel Lookup, or a vendor that failed a scenario).
• cpu = average CPU% across scenarios; mem = average peak memory (MB) across scenarios; perCpu = throughput per CPU% — the same values as the "Resource Efficiency (Across Scenarios)" table. Use null if not measurable.
• After writing the file, re-read it and confirm it parses as JSON and every avg/peak array has exactly 4 elements. Fix and rewrite if not.
• The GitHub Pages publish workflow overrides runId/date with the release tag, so approximate values there are fine, but keep them present.