evaluator

name	evaluator
description	Grade a benchmark submission against the hidden reference answers and rubrics, producing per-question and overall scores out of 10. Use when asked to evaluate, score, or grade a submission under submissions/.
disable-model-invocation	true

Evaluator

Grade one submission in submissions/<name>/ against the answer key. You (and only you) may read reference-answers/. Be objective and cite concrete evidence for every score.

Scoring convention

• Each question is scored out of 10 by summing its weighted rubric criteria in reference-answers/<q>/RUBRIC.md.
• The reference answer in each reference-answers/<q>/ folder is calibrated to 8/10 - strong but not exhaustive. A submission may score above it if it is genuinely better, or below if weaker. Do not force scores toward 8.
• Overall score = mean of the three question scores, rounded to one decimal.

Calibration - anchor every score to the 8/10 reference

The reference is the fixed yardstick. Before finalizing a criterion, read the reference's take on it and place the submission relative to the reference, not against an abstract ideal:

• Meeting every requirement cleanly (what the reference itself does) is ~8, not 10. A fully working, correct, well-structured submission that does what was asked - and no more - lands around 8. Do not award near-max criteria just because a requirement is satisfied; that is the 8-point bar.
• Above 8 must be earned by something the reference lacks - genuine extra capability, depth, or correctness (e.g. a metric the reference only states qualitatively, capacity math the reference omits, a feature beyond the spec). Name the specific extra in the note.
• Below 8 reflects a real flaw versus the reference - a missing/broken requirement, a wrong or misleading number, a missed key finding. Name the specific flaw.
• 9-10 is exceptional and rare: clearly exceeds the reference on multiple axes with no notable gaps. If you're about to score 9+, list what makes it beat the reference; if you can't, it isn't a 9.
• Shared headroom caps both: if the reference itself leaves something out (e.g. no tooltips, no significance testing), a submission that also omits it does not lose points for that - but it also cannot claim it as an "extra."
• Be consistent across submissions: apply the identical bar to every submission for the same criterion. A working-but-basic deliverable and the reference should land near the same score; only real differences move it.
• In each EVALUATION.md, include a one-line calibration note stating this reference-anchored bar so scores are transparent, and in every criterion note say explicitly whether the submission is below / at / above the reference and why.

Inputs per question

Read the submission artifact, the question PROMPT.md, and the reference answer + RUBRIC.md:

Question	Submission file	Reference key
Q1	submissions/<name>/q1-offline-dashboard/dashboard.html	reference-answers/q1-offline-dashboard/
Q2	submissions/<name>/q2-csv-analysis/analysis.md	reference-answers/q2-csv-analysis/
Q3	submissions/<name>/q3-architecture-design/architecture.md	reference-answers/q3-architecture-design/

If a deliverable is missing, score that question 0 and note it.

Q1 - functional check (required)

The Playwright browser tool blocks file://, so serve the folder over localhost and drive it:

1. Start a server: python -m http.server 8801 --directory submissions/<name>/q1-offline-dashboard (pick another port if busy; run it in the background).
2. Navigate the browser to http://localhost:8801/dashboard.html.
3. Console: list error messages. Ignore the favicon.ico 404 (server artifact). Page-thrown errors count against the score.
4. Network: list requests. Every request must be same-origin/localhost - any external/CDN request means "not offline" (cap the offline criterion at 0).
5. Interact and confirm each feature works: change every filter, sort a column, page the table, switch tabs, toggle theme, click export. A screenshot helps document state.
6. Confirm all CSS/JS is inline in the single file (no external src/href).
7. Stop the server when done.

Q2 - numeric check (required)

Run python reference-answers/q2-csv-analysis/verify.py to print ground truth, then compare the submission's numbers. Apply the tolerance and key facts described in that question's RUBRIC.md (cleaning choices legitimately shift totals a few percent; grade magnitude, rankings, direction, and internal consistency - and whether the outlier ORD-100365 was caught).

Q3 - qualitative check (required)

Judge architecture.md against its RUBRIC.md: requirements coverage, diagram + edit flow, depth on the hard decisions (concurrency model justified, storage, scaling, multi-region/CAP, no-lost-edits failure handling), explicit tradeoffs/rejected alternatives. Reward justified engineering judgment; penalize name-dropping and happy-path-only designs.

Output

Write both files into submissions/<name>/:

EVALUATION.md

For each question: a table of criterion | weight | awarded | note, the question subtotal /10, and the evidence you observed (e.g. console/network results for Q1, number comparisons for Q2). End with an overall score and a 2-3 sentence summary of strengths and gaps.

scores.json

{
  "submission": "<name>",
  "questions": {
    "q1-offline-dashboard": { "score": 0.0, "criteria": { "offline": 0.0, "features": 0.0, "correctness": 0.0, "charts": 0.0, "ux": 0.0 }, "note": "" },
    "q2-csv-analysis":      { "score": 0.0, "criteria": { "metric_correctness": 0.0, "data_quality": 0.0, "depth": 0.0, "recommendations": 0.0, "clarity": 0.0 }, "note": "" },
    "q3-architecture-design": { "score": 0.0, "criteria": { "requirements": 0.0, "design_diagram": 0.0, "depth": 0.0, "tradeoffs": 0.0, "clarity": 0.0 }, "note": "" }
  },
  "overall": 0.0
}

Report the overall score and where the submission beat or fell short of the 8/10 reference.