eval-corpus

name

eval-corpus

description

Use this skill when working with the faultline-eval evaluation harness: ingesting a labelled dataset into a normalised corpus, running coverage measurement against faultline playbooks, producing a coverage report, identifying high-value gap clusters, comparing baseline versus current results in CI, and generating playbook stubs from gap data. Trigger it for requests about running the eval pipeline, measuring playbook coverage, finding gaps, or promoting eval results as baselines.

Eval Corpus

This skill is for running faultline-eval, the deterministic log evaluation harness found in tools/eval-corpus/. Use it whenever the task involves:

• Ingesting a raw dataset (CSV/JSONL) into a normalised fixture corpus
• Running faultline against a corpus to measure playbook coverage
• Generating coverage reports (text, JSON, Markdown)
• Identifying unmatched failure clusters and generating playbook stubs
• Comparing baseline vs. current results as a CI regression gate
• Generating badge artifacts for README embedding

Do not use it for general playbook authoring, fixture ingestion via faultline fixtures ingest, or the internal real-fixture regression gate (make fixture-check). Those workflows use different tooling.

Read First

• SYSTEM.md
• tools/eval-corpus/README.md
• docs/playbooks.md
• The dataset config YAML in fixtures/datasets/ that you're working with

Pipeline Overview

ingest → manifest → run → report → gaps → compare → badge

Each subcommand is independent. Run them in order for a full evaluation.

Build

faultline-eval is a separate binary from the main faultline CLI. Build it once before running any subcommand:

go build -o bin/faultline-eval ./tools/eval-corpus

Or use the Makefile target:

make eval-build

Subcommand Reference

ingest

Reads a raw CSV (or JSONL) dataset and writes a normalised JSONL corpus.

bin/faultline-eval ingest \
  --config fixtures/datasets/<config>.yaml \
  --out    eval-work/corpus.jsonl

Required config fields:

input:
  type: csv
  path: fixtures/datasets/<file>.csv

parsing:
  log_field: <column-name>   # column containing the log/error text
  id_field: <column-name>    # stable row identifier (optional)
  timestamp_field: <column>  # event timestamp (optional)

processing:
  dedupe: true
  max_log_size: 200kb

run

Runs faultline detection on every fixture and writes results.

bin/faultline-eval run \
  --corpus  eval-work/corpus.jsonl \
  --out     eval-work/results.jsonl \
  --workers 8

report

Summarises coverage from a results file.

bin/faultline-eval report --results eval-work/results.jsonl
bin/faultline-eval report --results eval-work/results.jsonl --markdown
bin/faultline-eval report --results eval-work/results.jsonl --json

Key metrics: overall match rate, per-playbook hit counts, top unmatched tags.

gaps

Clusters unmatched fixtures and generates playbook stubs.

bin/faultline-eval gaps \
  --results     eval-work/results.jsonl \
  --fixtures    eval-work/corpus.jsonl \
  --out         eval-work/gaps \
  --max-samples 5

Output:

• eval-work/gaps/clusters.jsonl — machine-readable cluster list
• eval-work/gaps/cluster-summary.md — human-readable summary
• eval-work/gaps/samples/<cluster>/ — representative log files
• eval-work/gaps/playbook-stubs/ — YAML stubs ready for review

compare

CI regression gate: fail if coverage drops below a threshold.

bin/faultline-eval compare \
  --baseline baseline-results.jsonl \
  --current  eval-work/results.jsonl \
  --out      eval-work/comparison \
  --fail-on-coverage-drop \
  --min-match-rate 0.65

badge

Writes a compact coverage summary for README embedding.

bin/faultline-eval badge \
  --results        eval-work/results.jsonl \
  --out            eval-work/coverage-summary \
  --corpus-version ci-v1

Working With Large Datasets

The Travis Torrent dataset (fixtures/datasets/final-2017-01-25.csv) is 3.4 GB with ~3.9 M rows. Pre-filter it before ingesting to keep run times reasonable:

# Extract header + failed rows only
head -1 fixtures/datasets/final-2017-01-25.csv > eval-work/failed.csv
awk -F',' 'NR>1 && $65=="\"failed\"" && $60!="\"\"" { print }' \
  fixtures/datasets/final-2017-01-25.csv >> eval-work/failed.csv

Column 65 is tr_status, column 60 is tr_log_tests_failed. Adjust if needed.

Use fixtures/datasets/travis-torrent-2017.yaml as the ingest config.

Promoting Gap Clusters Into Playbooks

After generating gap cluster data, use the playbook-gap-authoring skill to close them systematically. That skill reads the cluster summary, triages each gap, extends or authors playbooks, runs linting and validation, and measures the coverage delta.

Hand off directly after producing eval-work/gaps/cluster-summary.md:

# Hand-off trigger for playbook-gap-authoring skill:
# "Close the top coverage gaps in eval-work/gaps/cluster-summary.md"

Do not manually edit playbook stubs from eval-work/gaps/playbook-stubs/. Those are starting-point scaffolding only — they require real sample verification and the full linter + make review gate before they are safe to commit.

Guardrails

• Always use dedupe: true to avoid inflating coverage metrics with duplicates.
• Set max_log_size to prevent extremely large log fields from skewing results.
• Use --fail-on-coverage-drop in CI to catch regressions early.
• Store the baseline results.jsonl as a CI artifact and download it at PR time.
• Never commit eval-work/ output — it is transient.