dbosk Canvas Quiz Authoring
Overview
Write and review Canvas LMS quizzes stored as INL1Quiz-<topic>.json files.
These quizzes use AllOrNothing scoring, require 100% to pass, allow unlimited
retakes with a 1-hour cooldown, and test conceptual understanding of
cryptography topics.
File Naming and Location
- Pattern:
modules/week-N/INL1Quiz-<topic>.json - Examples:
INL1Quiz-ciphers.json,INL1Quiz-zkp.json,INL1Quiz-mpc.json
Up-to-Date Format Reference
The canvaslms CLI provides canonical, up-to-date JSON examples. Run these
to get the latest format (always in sync with the tool):
canvaslms quizzes create --example # Full quiz envelope + all settings
canvaslms quizzes items add --example # All question types with examples
These show every supported question type (choice, multi-answer, matching,
true-false, ordering, rich-fill-blank, essay, file-upload,
formula) with correct scoring_data structure for each.
JSON Structure
Quiz Envelope
{
"quiz_type": "new",
"settings": {
"title": "INL1Quiz <Topic Name>",
"instructions": "<HTML instructions>",
"...other settings..."
},
"items": [ ]
}
Copy settings verbatim from an existing quiz file. Only change
settings.title. For the canonical settings block, see
references/quiz-settings.json.
Multi-Answer Question (primary format)
{
"position": 1,
"points_possible": 1.0,
"entry": {
"title": "Short Descriptive Title",
"item_body": "<p>Check all statements that are true about [topic].</p>",
"interaction_type_slug": "multi-answer",
"scoring_algorithm": "AllOrNothing",
"properties": {
"shuffle_rules": {
"choices": {
"to_lock": [],
"shuffled": false
}
}
},
"interaction_data": {
"choices": [
{
"position": 1,
"item_body": "<p>Statement text here.</p>"
}
]
},
"scoring_data": {
"value": [1, 3, 5]
}
}
}
Critical: scoring_data.value lists position numbers (1-indexed)
of correct choices. Every value must match a choice position in
interaction_data.choices.
Matching Question (for format variety)
{
"position": 1,
"points_possible": 1.0,
"entry": {
"title": "Match Properties",
"item_body": "<div>Match terms with definitions.</div>",
"interaction_type_slug": "matching",
"scoring_algorithm": "DeepEquals",
"properties": {
"shuffle_rules": {
"questions": { "shuffled": false }
}
},
"interaction_data": {
"answers": ["Definition A", "Definition B"],
"questions": [
{ "item_body": "Term A" },
{ "item_body": "Term B" }
]
},
"scoring_data": {
"value": {
"<uuid-for-term-A>": "Definition A",
"<uuid-for-term-B>": "Definition B"
},
"edit_data": {
"matches": [
{
"answer_body": "Definition A",
"question_id": "<uuid-for-term-A>",
"question_body": "Term A"
}
],
"distractors": []
}
}
}
}
Generate fresh v4 UUIDs for each question_id. Ensure scoring_data.value
maps each UUID to the correct answer and edit_data.matches mirrors the
mapping.
Question Design Principles
Target 6-8 Questions Per Quiz
Each quiz should have 6-8 questions covering different aspects of the topic.
Cognitive Levels (aim for a mix)
- Definitional: Match terms with definitions
- Mechanical: Protocol steps, equations, verification
- Conceptual: What a property guarantees, implications
- Applied: Concrete scenario, what happens?
- Analytical: Why does X work / not work?
Distractor Design
False choices must target specific misconceptions:
- "Longer keys prevent side-channel attacks" (confuses algorithmic vs implementation security)
- "Semi-honest security implies malicious security" (wrong adversary model relationship)
- "Knowledge extraction contradicts zero-knowledge" (confuses who has rewinding power)
Avoid obviously wrong distractors like "MPC can only compute simple functions."
Worked Example: Scenario-Based Distractors
The "Partial CSPRNG compromise" question from INL1Quiz-randomness.json
illustrates targeted distractor design for an applied-level question:
Stem: Assume a CSPRNG is partially compromised such that 96 bits of the 128-bit internal state is leaked. Which of these is/are true:
- "The CSPRNG remains secure, due to the forward security inherent to all CSPRNGs." — False. Targets the misconception that CSPRNGs inherently provide forward security; forward security is about protecting past outputs after state compromise, not preventing current state exploitation.
- "The CSPRNG is immediately compromised." — False. Targets overreaction: 32 bits remain unknown, so the state is not fully determined — but it is brute-forceable.
- "The CSPRNG can be compromised by a brute force attack with O(2^32) operations." — True. 128 − 96 = 32 unknown bits.
- "The CSPRNG can be compromised by a brute force attack with O(2^96) operations." — False. Targets the most common student error: confusing the leaked bits (96) with the remaining bits (32).
- "None of the above." — False. Safety distractor forcing active evaluation of every option rather than pattern-matching.
Notes:
- This uses
multi-answerwithscoring_data.value: [3](single correct choice). This is valid —multi-answer+AllOrNothingworks for both single- and multiple-correct questions. - The "is/are" phrasing in the stem avoids revealing how many choices are correct.
True/False Ratio
Aim for 55-75% true choices per question (e.g., 5 true out of 8 choices). Avoids "default to true" strategy (>80%) and "everything is a trick" feeling (<40%).
Contrast Pairs (Variation Theory)
Include choice pairs differing in one critical aspect:
- TRUE: "Privacy guarantees that aside from the output, no additional information is revealed"
- FALSE: "Privacy guarantees that no information about inputs is revealed, including through the output"
The critical aspect: privacy is relative to the output, not absolute.
Stems Must Not Point Out the Critical Aspect (Variation Theory)
When the quiz measures whether students discern an aspect, the stem must not name that aspect — the assessment principle of variation theory (Marton, Necessary Conditions of Learning, ch. 4: "the questions should not point out the relevant aspects of the problem to be solved, as this is exactly what the students are supposed to find out (discern)", p. 91; see also p. 89).
- BAD stem: "In the program below, the same five lines appear three times — once per room. What is the best improvement?" (the stem discerns the repetition for the student, who then only acts on it)
- GOOD stem: "What is the best improvement of the program below?" (seeing the repetition is the task)
In practice:
- Present phenomena and behaviours — a printout to predict, a crash to explain, wrong output where something else was expected — never the cause or the aspect behind them.
- Choices necessarily name aspects; there the student's task becomes judging the named claims against the code. Prefer distractors that mistake the aspect (comment the repeated blocks, rename variables per block) over transparently silly ones — this composes with the misconception rule under Distractor Design: the stem hides the aspect, each distractor embodies a wrong discernment of it.
- Watch for leaks via helpful phrasing: "which line is clearest for the user" names the usability aspect; "which line is the best option" does not.
Matching Distractors
Matching questions can include extra answers that don't match any term. Add
them to the answers array and the distractors list in edit_data:
"interaction_data": {
"answers": ["Stockholm", "Oslo", "Copenhagen", "Helsinki", "Berlin"],
"questions": [
{ "id": "q1", "item_body": "Sweden" },
{ "id": "q2", "item_body": "Norway" }
]
},
"scoring_data": {
"edit_data": {
"matches": [ ... ],
"distractors": ["Berlin"]
}
}
Format Variety
Include at least one non-multi-answer question per quiz. Available types:
multi-answerwithAllOrNothing— "select all true" (primary)matchingwithDeepEquals— match terms to definitionschoicewithEquivalence— single correct answer from choicestrue-falsewithEquivalence— true/false statementorderingwithDeepEquals— arrange items in correct order
Run canvaslms quizzes items add --example for JSON examples of each type.
Redundancy Analysis
Within a Quiz
- Do multiple questions test the same definitional knowledge in different formats? (Redundant unless testing genuinely different aspects.)
- Is the same property tested in more than two questions? (Likely redundant.)
- Do questions describe the same protocol steps? (Acceptable only if testing different aspects: commitment properties vs protocol flow vs simulation.)
Across Quizzes (students take all quizzes in a week)
- Does a quiz include content from another topic's quiz? (e.g., ZKPK properties in an MPC quiz — redundant.)
- Are the same examples reused without new insight?
Acceptable Overlap
- Different cognitive levels (definition vs application)
- Different protocol aspects (commitment vs verification vs simulation)
- Progression (general concept early, specific nuance later)
Validation
After editing quiz JSON, always run the validation script:
python3 ~/.claude/skills/canvas-quiz/scripts/validate_quiz.py <quiz-file.json>
To validate multiple files:
python3 ~/.claude/skills/canvas-quiz/scripts/validate_quiz.py modules/week-3/INL1Quiz-*.json
The script checks: valid JSON, required fields, scoring data consistency, true/false ratios, and prints a per-question summary.
Canvas Push Workflow
After editing:
cd modules/week-N
make push-quizzes-questions # Push with question replacement
make force-push-quizzes-questions # Force push (ignores timestamps)
Resources
| File | Content |
|---|---|
scripts/validate_quiz.py |
Validates quiz JSON structure and scoring data |
references/quiz-settings.json |
Canonical quiz_settings block to reuse |