| name | canvas-quiz |
| description | Write and review Canvas LMS quiz JSON files (INL1Quiz-*.json) for the tilkry
cryptography course. Use proactively when: (1) creating, editing, or
reviewing INL1Quiz JSON files, (2) user asks to write quiz questions for a
lecture topic, (3) user asks to review quiz quality, redundancy, or distractor
balance, (4) user mentions Canvas quiz, INL1Quiz, quiz JSON, or quiz
questions. Covers JSON structure, question design, scoring, redundancy
analysis, and validation.
|
Canvas Quiz Authoring
Overview
Write and review Canvas LMS quizzes stored as INL1Quiz-<topic>.json files.
These quizzes use AllOrNothing scoring, require 100% to pass, allow unlimited
retakes with a 1-hour cooldown, and test conceptual understanding of
cryptography topics.
File Naming and Location
- Pattern:
modules/week-N/INL1Quiz-<topic>.json
- Examples:
INL1Quiz-ciphers.json, INL1Quiz-zkp.json, INL1Quiz-mpc.json
Up-to-Date Format Reference
The canvaslms CLI provides canonical, up-to-date JSON examples. Run these
to get the latest format (always in sync with the tool):
canvaslms quizzes create --example
canvaslms quizzes items add --example
These show every supported question type (choice, multi-answer, matching,
true-false, ordering, rich-fill-blank, essay, file-upload,
formula) with correct scoring_data structure for each.
JSON Structure
Quiz Envelope
{
"quiz_type": "new",
"settings": {
"title": "INL1Quiz <Topic Name>",
"instructions": "<HTML instructions>",
"...other settings..."
},
"items": [ ]
}
Copy settings verbatim from an existing quiz file. Only change
settings.title. For the canonical settings block, see
references/quiz-settings.json.
Multi-Answer Question (primary format)
{
"position": 1,
"points_possible": 1.0,
"entry": {
"title": "Short Descriptive Title",
"item_body": "<p>Check all statements that are true about [topic].</p>",
"interaction_type_slug": "multi-answer",
"scoring_algorithm": "AllOrNothing",
"properties": {
"shuffle_rules": {
"choices": {
"to_lock": [],
"shuffled": false
}
}
},
"interaction_data": {
"choices": [
{
"position": 1,
"item_body": "<p>Statement text here.</p>"
}
]
},
"scoring_data": {
"value": [1, 3, 5]
}
}
}
Critical: scoring_data.value lists position numbers (1-indexed)
of correct choices. Every value must match a choice position in
interaction_data.choices.
Matching Question (for format variety)
{
"position": 1,
"points_possible": 1.0,
"entry": {
"title": "Match Properties",
"item_body": "<div>Match terms with definitions.</div>",
"interaction_type_slug": "matching",
"scoring_algorithm": "DeepEquals",
"properties": {
"shuffle_rules": {
"questions": { "shuffled": false }
}
},
"interaction_data": {
"answers": ["Definition A", "Definition B"],
"questions": [
{ "item_body": "Term A" },
{ "item_body": "Term B" }
]
},
"scoring_data": {
"value": {
"<uuid-for-term-A>": "Definition A",
"<uuid-for-term-B>": "Definition B"
},
"edit_data": {
"matches": [
{
"answer_body": "Definition A",
"question_id": "<uuid-for-term-A>",
"question_body": "Term A"
}
],
"distractors": []
}
}
}
}
Generate fresh v4 UUIDs for each question_id. Ensure scoring_data.value
maps each UUID to the correct answer and edit_data.matches mirrors the
mapping.
Question Design Principles
Target 6-8 Questions Per Quiz
Each quiz should have 6-8 questions covering different aspects of the topic.
Cognitive Levels (aim for a mix)
- Definitional: Match terms with definitions
- Mechanical: Protocol steps, equations, verification
- Conceptual: What a property guarantees, implications
- Applied: Concrete scenario, what happens?
- Analytical: Why does X work / not work?
Distractor Design
False choices must target specific misconceptions:
- "Longer keys prevent side-channel attacks" (confuses algorithmic vs
implementation security)
- "Semi-honest security implies malicious security" (wrong adversary model
relationship)
- "Knowledge extraction contradicts zero-knowledge" (confuses who has
rewinding power)
Avoid obviously wrong distractors like "MPC can only compute simple
functions."
Worked Example: Scenario-Based Distractors
The "Partial CSPRNG compromise" question from INL1Quiz-randomness.json
illustrates targeted distractor design for an applied-level question:
Stem: Assume a CSPRNG is partially compromised such that 96 bits of
the 128-bit internal state is leaked. Which of these is/are true:
- "The CSPRNG remains secure, due to the forward security inherent to
all CSPRNGs." — False. Targets the misconception that CSPRNGs
inherently provide forward security; forward security is about
protecting past outputs after state compromise, not preventing
current state exploitation.
- "The CSPRNG is immediately compromised." — False. Targets
overreaction: 32 bits remain unknown, so the state is not fully
determined — but it is brute-forceable.
- "The CSPRNG can be compromised by a brute force attack with
O(2^32) operations." — True. 128 − 96 = 32 unknown bits.
- "The CSPRNG can be compromised by a brute force attack with
O(2^96) operations." — False. Targets the most common student
error: confusing the leaked bits (96) with the remaining bits
(32).
- "None of the above." — False. Safety distractor forcing active
evaluation of every option rather than pattern-matching.
Notes:
- This uses
multi-answer with scoring_data.value: [3] (single correct
choice). This is valid — multi-answer + AllOrNothing works for both
single- and multiple-correct questions.
- The "is/are" phrasing in the stem avoids revealing how many choices are
correct.
True/False Ratio
Aim for 55-75% true choices per question (e.g., 5 true out of 8 choices).
Avoids "default to true" strategy (>80%) and "everything is a trick" feeling
(<40%).
Contrast Pairs (Variation Theory)
Include choice pairs differing in one critical aspect:
- TRUE: "Privacy guarantees that aside from the output, no additional
information is revealed"
- FALSE: "Privacy guarantees that no information about inputs is revealed,
including through the output"
The critical aspect: privacy is relative to the output, not absolute.
Stems Must Not Point Out the Critical Aspect (Variation Theory)
When the quiz measures whether students discern an aspect, the stem must
not name that aspect — the assessment principle of variation theory
(Marton, Necessary Conditions of Learning, ch. 4: "the questions should
not point out the relevant aspects of the problem to be solved, as this is
exactly what the students are supposed to find out (discern)", p. 91; see
also p. 89).
- BAD stem: "In the program below, the same five lines appear three
times — once per room. What is the best improvement?" (the stem discerns
the repetition for the student, who then only acts on it)
- GOOD stem: "What is the best improvement of the program below?"
(seeing the repetition is the task)
In practice:
- Present phenomena and behaviours — a printout to predict, a crash to
explain, wrong output where something else was expected — never the
cause or the aspect behind them.
- Choices necessarily name aspects; there the student's task becomes
judging the named claims against the code. Prefer distractors that
mistake the aspect (comment the repeated blocks, rename variables per
block) over transparently silly ones — this composes with the
misconception rule under Distractor Design: the stem hides the aspect,
each distractor embodies a wrong discernment of it.
- Watch for leaks via helpful phrasing: "which line is clearest for the
user" names the usability aspect; "which line is the best option" does
not.
Matching Distractors
Matching questions can include extra answers that don't match any term. Add
them to the answers array and the distractors list in edit_data:
"interaction_data": {
"answers": ["Stockholm", "Oslo", "Copenhagen", "Helsinki", "Berlin"],
"questions": [
{ "id": "q1", "item_body": "Sweden" },
{ "id": "q2", "item_body": "Norway" }
]
},
"scoring_data": {
"edit_data": {
"matches": [ ... ],
"distractors": ["Berlin"]
}
}
Format Variety
Include at least one non-multi-answer question per quiz. Available types:
multi-answer with AllOrNothing — "select all true" (primary)matching with DeepEquals — match terms to definitionschoice with Equivalence — single correct answer from choicestrue-false with Equivalence — true/false statementordering with DeepEquals — arrange items in correct order
Run canvaslms quizzes items add --example for JSON examples of each type.
Redundancy Analysis
Within a Quiz
- Do multiple questions test the same definitional knowledge in different
formats? (Redundant unless testing genuinely different aspects.)
- Is the same property tested in more than two questions? (Likely redundant.)
- Do questions describe the same protocol steps? (Acceptable only if testing
different aspects: commitment properties vs protocol flow vs simulation.)
Across Quizzes (students take all quizzes in a week)
- Does a quiz include content from another topic's quiz? (e.g., ZKPK
properties in an MPC quiz — redundant.)
- Are the same examples reused without new insight?
Acceptable Overlap
- Different cognitive levels (definition vs application)
- Different protocol aspects (commitment vs verification vs simulation)
- Progression (general concept early, specific nuance later)
Validation
After editing quiz JSON, always run the validation script:
python3 ~/.claude/skills/canvas-quiz/scripts/validate_quiz.py <quiz-file.json>
To validate multiple files:
python3 ~/.claude/skills/canvas-quiz/scripts/validate_quiz.py modules/week-3/INL1Quiz-*.json
The script checks: valid JSON, required fields, scoring data consistency,
true/false ratios, and prints a per-question summary.
Canvas Push Workflow
After editing:
cd modules/week-N
make push-quizzes-questions
make force-push-quizzes-questions
Resources
| File | Content |
|---|
scripts/validate_quiz.py | Validates quiz JSON structure and scoring data |
references/quiz-settings.json | Canonical quiz_settings block to reuse |
