| name | test-taker-da-corrective |
| description | Take the data-analyst corrective (find-the-mistakes) capability benchmark - create a submission folder named <llm>+<harness> under submissions/da-corrective/ and answer all nine corrective questions across three types (fix a broken HTML dashboard, find the wrong claims in a CSV analysis, find the engineering mistakes in an architecture design), each at easy/mid/hard. Use when asked to take the da-corrective test, sit the da-corrective benchmark, or generate a da-corrective submission. |
| disable-model-invocation | true |
Test Taker (DA Corrective)
You are sitting the da-corrective (data-analyst) capability benchmark. Each question gives you an existing artifact that is mostly correct but contains injected mistakes. Your job is to find / fix them by working out the ground truth, not by building from scratch. Work independently and do your best.
Rules (read before starting)
- Read only from
questions/da-corrective/<question>/. Never open, read, or reference reference-answers/ - it holds the answer keys (clean answers, HTML checkers, TRUTH files, MISTAKES keys), and consulting it invalidates the run.
- Write only inside your own submission folder. Do not modify
questions/, other submissions, or anything else in the repo.
- Budget roughly 6 minutes per question. Higher difficulty = more and subtler mistakes.
Step 1 - Create your submission folder
Name it <llm>+<harness> in lowercase with no spaces. Examples: composer2.5+cursor, gpt-5+claude-code, sonnet+opencode. If you cannot determine your model or harness, ask the operator once, then proceed.
submissions/da-corrective/<llm>+<harness>/
q1-dashboard-easy/ q1-dashboard-mid/ q1-dashboard-hard/
q2-csv-easy/ q2-csv-mid/ q2-csv-hard/
q3-arch-easy/ q3-arch-mid/ q3-arch-hard/
Step 2 - Answer each question
Read each PROMPT.md and the corrupted artifact in that question's folder, then write the exact deliverable into the matching subfolder of your submission. The deliverable depends on the question type:
| Question | Read in the question folder | Produce in your submission | What you are doing |
|---|
q1-dashboard-* | PROMPT.md and dashboard.html | dashboard.html (the fixed file) | the broken single-file offline dashboard has bugs that break rendering/running; fix it so it fully runs |
q2-csv-* | PROMPT.md, sales_data.csv, analysis.md | corrections.json | the analysis report is annotated with labeled claims [C1]..[Cn]; recompute from the CSV and return only the labels you believe are wrong, mapped to the correct value |
q3-arch-* | PROMPT.md and architecture.md | mistakes.md | the design doc has engineering mistakes; list each as a numbered entry (anchor/quote, what is wrong, why it matters, the correct design) |
Per-type details
- Q1 (dashboard): keep exactly one file
dashboard.html, fully self-contained: all CSS/JS inline, no external/CDN resources, no network requests, no build step. It must run when served over http://localhost and every required feature must work end-to-end (KPIs recompute on filters; hand-drawn line/bar/pie canvas charts draw; region/category/month/search filters live-update KPIs+charts+table; table is sortable+searchable+paginated; tabbed views switch; theme toggle visibly changes the theme; CSV export downloads a file). Scoring is binary: 10 if a headless browser checker says the dashboard fully runs with no console errors/features broken, else 0 - so fix everything, not just the first bug.
- Q2 (CSV): the corrupted
analysis.md lists every checkable claim with a tag like `[C1]`. Compute the ground truth directly from sales_data.csv (cleaning the same way the report says it cleaned it). Return a corrections.json mapping only the labels you believe are wrong to the correct value, e.g. {"C6": "North", "C11": "positive", "C14": "ORD-300012"}. Numbers may be numbers or strings; strings match case-insensitively (region/category/id/sign). Including a label that was actually correct is a false positive; giving the wrong corrected value for a genuinely wrong claim is also a miss. Tolerance: ~0.5% on money totals/AOV, 0.5 percentage points on rates, exact for rankings/ids/sign. Mid and hard require recomputing several interlocking metrics (profit, correlation, cross-cuts, and - hard only - currency conversion and YoY), so a programmatic approach is expected.
- Q3 (architecture): real engineering mistakes only - decisions that contradict a stated requirement, named-but-misapplied tech, lost-edits paths, split-brain/consistency holes, scaling contradictions, security gaps. Style/brevity or a justified tradeoff you'd simply make differently is not a mistake. Every entry you list is scored: correct finds help; unsupported entries hurt.
Step 3 - Finish
Verify all nine deliverables exist at the correct paths (3 × dashboard.html for Q1, 3 × corrections.json for Q2, 3 × mistakes.md for Q3), then report your submission folder name. Do not grade yourself.
