e2e-observability

# E2E Observability Structured tracking of E2E test results. Replaces freeform markdown reports with validated JSON that's queryable, aggregatable, and consistent across runs. ## Directory Structure All data lives in the repo at `<working-dir>/e2e-results/`: ``` e2e-results/ ├── taxonomy.json # Failure categories (human-curated, agent-enforced) ├── index.json # Auto-generated aggregation (never edit manually) ├── benchmarks/ # Per-ATS-provider history (auto-generated) │ └── <provider>-<site>.json └── runs/ # One file per E2E run (agent writes this) └── YYYY-MM-DD-HH-MM.json ``` Working dir: `<REPO_ROOT>` ## Workflow ### After each E2E run: 1. Write `e2e-results/runs/YYYY-MM-DD-HH-MM.json` using the schema below 2. Run the validator: `python3 e2e-results/scripts/validate_and_aggregate.py` 3. If validation fails: fix the JSON and rerun 4. The script auto-updates `benchmarks/` and `index.json` 5. Commit all changes to git ### To check trends (before selecting issues): Read `e2e-results/index.json` — it has: - Per-provider success rates - Top failure categories (sorted by count) - Which failures are code bugs vs external blockers Pick the highest-impact code bug to fix next. ## Run JSON Schema Each E2E run produces one file: `runs/YYYY-MM-DD-HH-MM.json` ```json { "timestamp": "2026-03-30T07:00:00+08:00", "test_id": "linkedin-ats-single | linkedin-batch-apply | ats-single | smoke", "iteration": 128, "duration_s": 960, "status": "pass | partial | fail | timeout", "score": {"met": 8, "total": 9, "pct": 89}, "ats_providers": [ { "provider": "greenhouse | lever | ashby | workday | icims | smartrecruiters | linkedin_easy_apply | other", "site": "SimpliSafe", "url": "https://...", "result": "pass | fail | skip", "failure": null, "fields_filled": 17, "fields_total": 17, "duration_s": 180, "notes": "" } ], "issues_found": [ { "category": "combobox_stuck", "provider": "greenhouse", "description": "Veteran Status dropdown: options visible but unselectable", "is_new": true } ] } ``` **Rules:** - `failure` must be a category ID from `taxonomy.json` or null if pass - `provider` must be one of the known providers or "other" - `fields_filled` and `fields_total` are required when result is "pass" or "fail" (not "skip") - `issues_found[].category` must match a `taxonomy.json` entry ## Taxonomy Read `references/taxonomy.json` for the current failure categories. Only a human should add new categories — if you encounter a failure that doesn't fit, use `"other"` and describe it in `notes`. The human will decide whether to add a new category. ## Reading Trends `index.json` is the entry point for understanding what's working and what isn't: ```json { "last_updated": "...", "total_runs": 33, "provider_scores": { "greenhouse": {"attempts": 8, "passes": 6, "rate": 75}, "workday": {"attempts": 5, "passes": 0, "rate": 0} }, "top_failures": [ {"category": "combobox_stuck", "count": 8, "is_code_bug": true}, {"category": "login_wall", "count": 4, "is_code_bug": false} ], "recent_runs": [...] } ``` Use this to prioritize: highest-count `is_code_bug: true` failure = most impactful fix.