| name | evaluator-da-constructive |
| description | Grade a data-analyst constructive (build-from-scratch) benchmark submission under submissions/da-constructive/ against the hidden reference answers and rubrics, producing per-question and overall scores out of 10. Use when asked to evaluate, score, or grade a da-constructive submission. |
| disable-model-invocation | true |
Evaluator (DA Constructive)
Grade one da-constructive submission in submissions/da-constructive/<name>/ against the answer key. You (and only you) may read reference-answers/. Be objective and cite concrete evidence for every score.
Scoring convention
- Each question is scored out of 10 by summing its weighted rubric criteria in
reference-answers/da-constructive/<q>/RUBRIC.md.
- The reference answer in each
reference-answers/da-constructive/<q>/ folder is calibrated to 8/10 - strong but not exhaustive. A submission may score above it if it is genuinely better, or below if weaker. Do not force scores toward 8.
- Overall score = mean of the three question scores, rounded to one decimal.
Calibration - anchor every score to the 8/10 reference
The reference is the fixed yardstick. Before finalizing a criterion, read the reference's take on it and place the submission relative to the reference, not against an abstract ideal:
- Meeting every requirement cleanly (what the reference itself does) is ~8, not 10. A fully working, correct, well-structured submission that does what was asked - and no more - lands around 8. Do not award near-max criteria just because a requirement is satisfied; that is the 8-point bar.
- Above 8 must be earned by something the reference lacks - genuine extra capability, depth, or correctness (e.g. a metric the reference only states qualitatively, capacity math the reference omits, a feature beyond the spec). Name the specific extra in the note.
- Below 8 reflects a real flaw versus the reference - a missing/broken requirement, a wrong or misleading number, a missed key finding. Name the specific flaw.
- 9-10 is exceptional and rare: clearly exceeds the reference on multiple axes with no notable gaps. If you're about to score 9+, list what makes it beat the reference; if you can't, it isn't a 9.
- Shared headroom caps both: if the reference itself leaves something out (e.g. no tooltips, no significance testing), a submission that also omits it does not lose points for that - but it also cannot claim it as an "extra."
- Be consistent across submissions: apply the identical bar to every submission for the same criterion. A working-but-basic deliverable and the reference should land near the same score; only real differences move it.
- In each
EVALUATION.md, include a one-line calibration note stating this reference-anchored bar so scores are transparent, and in every criterion note say explicitly whether the submission is below / at / above the reference and why.
Inputs per question
Read the submission artifact, the question PROMPT.md, and the reference answer + RUBRIC.md:
| Question | Submission file | Reference key |
|---|
| Q1 | submissions/da-constructive/<name>/q1-offline-dashboard/dashboard.html | reference-answers/da-constructive/q1-offline-dashboard/ |
| Q2 | submissions/da-constructive/<name>/q2-csv-analysis/analysis.md | reference-answers/da-constructive/q2-csv-analysis/ |
| Q3 | submissions/da-constructive/<name>/q3-architecture-design/architecture.md | reference-answers/da-constructive/q3-architecture-design/ |
If a deliverable is missing, score that question 0 and note it.
Q1 - functional check (required)
The Playwright browser tool blocks file://, so serve the folder over localhost and drive it:
- Start a server:
python -m http.server 8801 --directory submissions/da-constructive/<name>/q1-offline-dashboard (pick another port if busy; run it in the background).
- Navigate the browser to
http://localhost:8801/dashboard.html.
- Console: list error messages. Ignore the
favicon.ico 404 (server artifact). Page-thrown errors count against the score.
- Network: list requests. Every request must be same-origin/localhost - any external/CDN request means "not offline" (cap the offline criterion at 0).
- Interact and confirm each feature works: change every filter, sort a column, page the table, switch tabs, toggle theme, click export. A screenshot helps document state.
- Confirm all CSS/JS is inline in the single file (no external
src/href).
- Stop the server when done.
Q2 - numeric check (required)
Run python reference-answers/da-constructive/q2-csv-analysis/verify.py to print ground truth, then compare the submission's numbers. Apply the tolerance and key facts described in that question's RUBRIC.md (cleaning choices legitimately shift totals a few percent; grade magnitude, rankings, direction, and internal consistency - and whether the outlier ORD-100365 was caught).
Q3 - qualitative check (required)
Judge architecture.md against its RUBRIC.md: requirements coverage, diagram + edit flow, depth on the hard decisions (concurrency model justified, storage, scaling, multi-region/CAP, no-lost-edits failure handling), explicit tradeoffs/rejected alternatives. Reward justified engineering judgment; penalize name-dropping and happy-path-only designs.
Output
Write both files into submissions/da-constructive/<name>/:
EVALUATION.md
For each question: a table of criterion | weight | awarded | note, the question subtotal /10, and the evidence you observed (e.g. console/network results for Q1, number comparisons for Q2). End with an overall score and a 2-3 sentence summary of strengths and gaps.
scores.json
{
"submission": "<name>",
"set": "da-constructive",
"questions": {
"q1-offline-dashboard": { "score": 0.0, "criteria": { "offline": 0.0, "features": 0.0, "correctness": 0.0, "charts": 0.0, "ux": 0.0 }, "note": "" },
"q2-csv-analysis": { "score": 0.0, "criteria": { "metric_correctness"
Report the overall score and where the submission beat or fell short of the 8/10 reference.
