e2e-observability

name

e2e-observability

description

Structured E2E test result tracking for the Browser Use extension heartbeat loop. After each E2E run, write a structured JSON result, validate it, and auto-aggregate per-provider benchmarks and failure trends. Use when: an E2E test completes and needs recording, checking E2E trends, querying "which ATS providers are failing", reviewing benchmark history, or when the heartbeat loop reports E2E results. Triggers on: "record E2E result", "E2E observability", "benchmark status", "ATS provider history", "failure trends", "what's failing", "E2E summary".

E2E Observability

Structured tracking of E2E test results. Replaces freeform markdown reports with validated JSON that's queryable, aggregatable, and consistent across runs.

Directory Structure

All data lives in the repo at <working-dir>/e2e-results/:

e2e-results/
├── taxonomy.json          # Failure categories (human-curated, agent-enforced)
├── index.json             # Auto-generated aggregation (never edit manually)
├── benchmarks/            # Per-ATS-provider history (auto-generated)
│   └── <provider>-<site>.json
└── runs/                  # One file per E2E run (agent writes this)
    └── YYYY-MM-DD-HH-MM.json

Working dir: <REPO_ROOT>

Workflow

After each E2E run:

1. Write e2e-results/runs/YYYY-MM-DD-HH-MM.json using the schema below
2. Run the validator: python3 e2e-results/scripts/validate_and_aggregate.py
3. If validation fails: fix the JSON and rerun
4. The script auto-updates benchmarks/ and index.json
5. Commit all changes to git

To check trends (before selecting issues):

Read e2e-results/index.json — it has:

• Per-provider success rates
• Top failure categories (sorted by count)
• Which failures are code bugs vs external blockers

Pick the highest-impact code bug to fix next.

Run JSON Schema

Each E2E run produces one file: runs/YYYY-MM-DD-HH-MM.json

{
  "timestamp": "2026-03-30T07:00:00+08:00",
  "test_id": "linkedin-ats-single | linkedin-batch-apply | ats-single | smoke",
  "iteration": 128,
  "duration_s": 960,
  "status": "pass | partial | fail | timeout",
  "score": {"met": 8, "total": 9, "pct": 89},
  "ats_providers": [
    {
      "provider": "greenhouse | lever | ashby | workday | icims | smartrecruiters | linkedin_easy_apply | other",
      "site": "SimpliSafe",
      "url": "https://...",
      "result": "pass | fail | skip",
      "failure": null,
      "fields_filled": 17,
      "fields_total": 17,
      "duration_s": 180,
      "notes": ""
    }
  ],
  "issues_found": [
    {
      "category": "combobox_stuck",
      "provider": "greenhouse",
      "description": "Veteran Status dropdown: options visible but unselectable",
      "is_new": true
    }
  ]
}

Rules:

• failure must be a category ID from taxonomy.json or null if pass
• provider must be one of the known providers or "other"
• fields_filled and fields_total are required when result is "pass" or "fail" (not "skip")
• issues_found[].category must match a taxonomy.json entry

Taxonomy

Read references/taxonomy.json for the current failure categories. Only a human should add new categories — if you encounter a failure that doesn't fit, use "other" and describe it in notes. The human will decide whether to add a new category.

Reading Trends

index.json is the entry point for understanding what's working and what isn't:

{
  "last_updated": "...",
  "total_runs": 33,
  "provider_scores": {
    "greenhouse": {"attempts": 8, "passes": 6, "rate": 75},
    "workday": {"attempts": 5, "passes": 0, "rate": 0}
  },
  "top_failures": [
    {"category": "combobox_stuck", "count": 8, "is_code_bug": true},
    {"category": "login_wall", "count": 4, "is_code_bug": false}
  ],
  "recent_runs": [...]
}

Use this to prioritize: highest-count is_code_bug: true failure = most impactful fix.