evaluator-da-corrective

name	evaluator-da-corrective
description	Grade a data-analyst corrective (find-the-mistakes) benchmark submission under submissions/da-corrective/ - run the scripted Q1 HTML checker (10/0), the scripted Q2 verify.py (net counting), and LLM-match the Q3 mistakes.md entries to the MISTAKES keys. Use when asked to evaluate, score, or Grade a da-corrective submission.
disable-model-invocation	true

Evaluator (DA Corrective)

Grade one da-corrective submission in submissions/da-corrective/<name>/. You (and only you) may read reference-answers/da-corrective/ - it holds the answer keys and the graders. Be objective and cite concrete evidence for every score. Per-question grading is largely scripted (Q1 and Q2) - run the scripts and record their output verbatim; only Q3 is LLM-graded.

Scoring convention (three flavors)

• Q1 (dashboard, 3 questions): binary. 10 if the question's headless Playwright checker exits 0 (the dashboard fully runs: no console errors, no external requests, KPIs/charts/table populated and live, filters/sort/pagination/tabs/theme/export all work); 0 otherwise. No partial credit.
• Q2 (CSV, 3 questions): net counting. Run the question's verify.py against the submitted corrections.json; it prints {correct_found, wrong_found, missed, net} where net = correct_found - wrong_found.
• Q3 (architecture, 3 questions): net counting via LLM matching (see below): net = correct_found - wrong_found.
• Overall: the sum across all nine questions (Q1 contributes 0-30; Q2/Q3 can go negative).

Inputs per question

Question	Submission file	Reference key
q1-dashboard-easy/mid/hard	submissions/da-corrective/<name>/q1-dashboard-*/dashboard.html	reference-answers/da-corrective/q1-dashboard-*/ (check.mjs, MISTAKES.md)
q2-csv-easy/mid/hard	submissions/da-corrective/<name>/q2-csv-*/corrections.json	reference-answers/da-corrective/q2-csv-*/ (verify.py, TRUTH.json, analysis.md)
q3-arch-easy/mid/hard	submissions/da-corrective/<name>/q3-arch-*/mistakes.md	reference-answers/da-corrective/q3-arch-*/ (MISTAKES.md, architecture.md)

If a deliverable is missing, record the question's score as 0 (Q1) or {correct_found:0,wrong_found:0,missed:N,net:0} (Q2/Q3 with missed:N).

One-time setup

• Q1 checker needs Playwright Chromium. From the repo root, install once: npm install (already pulls playwright) then npx playwright install chromium.

Q1 - run the scripted checker (required, no LLM grading)

For each Q1 question, run that question's checker pointing at the submitted dashboard.html (an absolute path works; the checker serves the file's own directory over localhost and drives headless Chromium):

node reference-answers/da-corrective/q1-dashboard-easy/check.mjs  <abs>/submissions/da-corrective/<name>/q1-dashboard-easy/dashboard.html

• Exit code 0 and stdout PASS -> score 10.
• Exit code non-zero and stdout FAIL (with one or more bulleted failure reasons) -> score 0; copy the failure reasons into your note.
• If a deliverable is missing -> 0.
• The checkers are intentionally strict (no browser judgment); do not relax them. Do not open the dashboard in a browser yourself - the checker is the source of truth.

Q2 - run the scripted verifier (required, no LLM grading)

For each Q2 question, run:

python reference-answers/da-corrective/q2-csv-easy/verify.py  <abs>/submissions/da-corrective/<name>/q2-csv-easy/corrections.json

• It prints a JSON object {"correct_found":N,"wrong_found":N,"missed":N,"net":N} (or a note for a missing/invalid file). Record it verbatim; the question net is the score.
• verify.py compares against TRUTH.json, which encodes the correct expected values and tolerance; there is no eyeballing of numbers. Note any note field (e.g. missing/invalid corrections.json).

Q3 - LLM-match the submitted mistakes.md (required)

For each Q3 question:

1. Read the submission's mistakes.md and the reference key reference-answers/da-corrective/q3-arch-*/MISTAKES.md (which lists the injected mistakes, each with a stable id, an anchor quote, a category, the wrong claim, and the correct reasoning). Optionally skim the corruption architecture.md to confirm an anchor.
2. Match each numbered entry in the submission to at most one ground-truth mistake id. A match requires BOTH: (a) the entry's anchor/quote points at the same place as the ground-truth mistake, AND (b) the entry identifies the same defect category (e.g. lost-edits-before-quorum, split-brain, ACL-not-per-op) - not a generic "this is underexplained".
3. Compute:
   • correct_found = number of distinct ground-truth ids matched by at least one submission entry.
   • wrong_found = submission entries that do NOT match any ground-truth id (non-issues, duplicates of an already-matched id, or unsupported/generic claims).
   • missed = ground-truth ids not matched (informational).
   • net = correct_found - wrong_found.

• Multiple submission entries mapping to the same id count once as correct_found; the extras count as wrong_found (no inflation by restating).
• Be strict: a submission entry that vaguely criticizes without naming the actual defect is NOT a match. Read the MISTAKES.md anchor + "What is wrong" to judge; the entry must essentially identify the same defect.
• Provide a per-id note: did the submission correctly find it? with what evidence? Or which entry was a non-issue?

Output

Write both files into submissions/da-corrective/<name>/:

EVALUATION.md

A per-question table. One row per question with columns:

• Q1 rows: question | kind=run | score(0/10) | check output (PASS/FAIL + first failure reason).
• Q2 rows: question | kind=net | correct_found | wrong_found | missed | net | verify.py note.
• Q3 rows: question | kind=net | correct_found | wrong_found | missed | net | per-id notes. End with the overall (sum across nine) and a 2-3 sentence summary of strengths and gaps (which Q1 features still broke, which Q2 metrics were misflagged, which Q3 categories were caught vs invented).

scores.json

{
  "submission": "<name>",
  "set": "da-corrective",
  "questions": {
    "q1-dashboard-easy":  { "kind": "run",  "score": 0, "note": "" },
    "q1-dashboard-mid":   { "kind": "run",  "score": 0, "note": "" },
    "q1-dashboard-hard":   { "kind": "run",  "score": 0, "note": "" },
                           
                            
                           
                          
                           
                          
  
   

overall = sum of the three Q1 scores + all six net values (can be lower than the Q1-only contribution if the net counts go negative).

Report the overall score and where the pair found or missed the injected mistakes.