| name | evaluator-fe-corrective |
| description | Grade a frontend corrective (debugging) benchmark submission under submissions/fe-corrective/ by running each question's Playwright check.mjs (binary 10/0). Use when asked to evaluate, score, or grade a fe-corrective submission. |
| disable-model-invocation | true |
Evaluator (FE Corrective)
Grade one fe-corrective submission in submissions/fe-corrective/<name>/. You (and only you) may read reference-answers/fe-corrective/. Per-question grading is fully scripted — run the checkers and record output; no LLM matching.
Scoring
- Each of the nine questions: 10 if
check.mjs exits 0 and prints PASS; 0 if FAIL or missing deliverable.
- Overall: sum across nine (max 90).
Inputs
| Question | Submission | Checker |
|---|
q1-web-easy/mid/hard | submissions/fe-corrective/<name>/q1-web-*/ | reference-answers/fe-corrective/q1-web-*/check.mjs |
q2-react-easy/mid/hard | submissions/fe-corrective/<name>/q2-react-*/ | reference-answers/fe-corrective/q2-react-*/check.mjs |
q3-app-easy/mid/hard | submissions/fe-corrective/<name>/q3-app-*/ | reference-answers/fe-corrective/q3-app-*/check.mjs |
Pass the submission directory (absolute path) as the sole argument to each checker.
One-time setup
From repo root: npm install then npx playwright install chromium (shared with da-corrective Q1).
Run checkers (required)
For each question:
node reference-answers/fe-corrective/q1-web-easy/check.mjs <abs>/submissions/fe-corrective/<name>/q1-web-easy
(Repeat for all nine ids.) Exit 0 + PASS → 10; non-zero + FAIL (+ bullets) → 0. Copy the first failure reason into the note. Do not relax checkers or grade in a manual browser.
Output
Write into submissions/fe-corrective/<name>/:
EVALUATION.md
Per-question table: question | kind=run | score(0/10) | check output. End with overall (sum) and a short summary.
scores.json
{
"submission": "<name>",
"set": "fe-corrective",
"questions": {
"q1-web-easy": { "kind": "run", "score": 0, "note": "" },
"q1-web-mid": { "kind": "run", "score": 0, "note": "" },
"q1-web-hard": { "kind": "run", "score": 0, "note": "" },
Report the overall score and which questions still failed.
