| name | sharpening-quiz-questions |
| description | Use when revising or authoring multiple-choice question banks in domain/src/*_questions/*.ts (RawQuestion format with a-f answers, c=correct, d=partial-credit) — especially when the correct answer is guessable from length, phrasing, or tone rather than knowledge, when questions read like a textbook quiz instead of a conversational self-assessment, or when distractors are all equally easy/hard to eliminate. |
Sharpening Quiz Questions
Overview
Multiple-choice question banks written in one pass tend to leak the answer through form rather than content: the correct answer is longer, more hedged, more complete-sounding; the "wrong" answers are uniformly implausible. A test-taker who has never seen the material can still score well by pattern-matching on style. This skill is a multi-pass revise loop that measures those tells objectively where possible, diagnoses which ones dominate a given file, and fixes only those — rather than doing a uniform rewrite that just relocates the tell somewhere else.
Confirmed baseline: running scripts/answer-length-stats.mjs against this repo's own software_testing_questions/*.ts files showed c and d averaging 2x the word count of the wrong answers, with every question flagged. This is the default failure mode — assume it's present until measured otherwise.
When to Use
- User says a quiz "feels gameable," "too easy to guess," or "like a math quiz not a self-assessment"
- You're about to write a new
RawQuestion[] tier file, or revise an existing one
- Spot-checking reveals the correct answer is always the most detailed/hedged option
Not for: single one-off questions (just write them carefully), or files that already pass the length-stats check with a good format mix.
Dimensions Checklist
Each pass, pick 2-3 dimensions with the worst signal in the file — don't try to fix everything at once, it produces shallow edits everywhere instead of real fixes anywhere.
| Dimension | Tell it produces | How to check |
|---|
| Answer length parity | Correct/partial answers are conspicuously longer or more hedged | scripts/answer-length-stats.mjs <file> — flags >25% deviation from the a/b/e/f average |
| Distractor plausibility split | All 4 wrong answers (a,b,e,f) are equally silly, or equally tempting | Read each wrong answer cold: 2-3 should be clean category errors (obviously wrong to anyone with basic exposure); 1-2 should be a real misconception or half-truth someone with partial knowledge would pick |
| Partial-credit (d) calibration | Every d uses the same formula ("shorter/simpler version of c") | Vary it: sometimes d is a defensible simplification close to c; sometimes it's clearly a notch below (misses the key nuance). If every question uses the same pattern, that pattern is the tell |
| Lexical hedging bias | Words like "usually/often/can/typically" cluster in correct answers; "always/never/only/completely" cluster in wrong ones | Skim a/b/e/f vs c/d for absolute-language clustering |
| Format monotony | Every stem is "What is X?" / "What does X mean?" | Track format per question; aim for a mix (see below) — pure recall should be well under half the file |
| Conversational tone | Stems read like textbook definitions, not something you'd ask a person | Rewrite stems as scenario/reaction/judgment prompts where the topic supports it |
Question format variety (mix across a 10-question tier, don't force every question into a novel format):
- Knowledge-recall — "What is X?" (keep some — self-assessments still need a factual floor)
- Scenario/situational — "You inherit a PR with zero tests and a tight deadline. What's your first move?"
- Diagnostic/spot-the-flaw — describe a test or approach, ask what's wrong with it
- Trade-off/dilemma — "You can only add one of these safeguards before ship. Which matters most here?"
- Reaction — "A teammate says '90% coverage means we're solid.' How do you respond?"
- What-would-you-ask-first — probes triage/investigation instinct rather than a fact
Process
- Diagnose. Run
node scripts/answer-length-stats.mjs <file> on the target file. Read 3-4 questions cold, asking: could I guess the answer without knowing the material? Note which dimensions above are failing.
- Pick this pass's focus. 2-3 dimensions, based on what's actually broken — not the full checklist every time.
- Revise. Rewrite only what the chosen dimensions require. Keep
mmr and tags and the underlying topic/difficulty intent unless they're wrong. Keep c as the correct slot and d as partial-credit — the schema is fixed, the content isn't (answer order is shuffled before display, so slot position itself isn't a leak — length and phrasing are).
- Verify. Re-run the length-stats script; confirm the flagged count dropped. Cold-read a sample again.
- Repeat for the next dimension set or move to the next file. Two to three passes per file is normal — one giant pass tends to under-fix everything.
- After all files in a directory are done, run this repo's standard domain check:
cd domain && npm test -- --run && npm run build && npm run lint to confirm nothing broke the RawQuestion shape (still needs a through f, mmr, and tags.length >= 2).
Common Mistakes
- Fixing length by padding the wrong answers instead of trimming c/d. Padding a/b/e/f with filler is the lazier fix but reads as artificial ("why does this wrong answer ramble"). Prefer tightening c/d to be precise, not just short.
- Making every wrong answer equally plausible. A self-assessment still needs some answers a knowledgeable person eliminates instantly — total ambiguity across all 6 options isn't "hard," it's just noisy.
- Rewriting the whole file in one pass. Produces surface-level fixes on every dimension instead of real fixes on the 2-3 that mattered most.
- Forgetting the schema constraint.
c must stay the fully-correct answer and d the partial-credit answer for TieredQuestioner.answer() scoring to keep working — vary content and tone, never the slot roles.
