Agentic Exam Prep

Agentic Exam Prep

Overview

An assembly line for open-book exam packs: an iterative MCQ drill loop that tightens recall and reveals cheatsheet gaps, a print-pack architecture, course-material mining, and cram-pod/PDF tooling. Course-agnostic in mechanics; the depth axes and sources are re-designed per course (two worked applications: the COMP4021 and COMP4431 arc repos in this org). The point is to drill 7–15 correctly-chosen questions, not 561 — selection and signal quality are the whole game.

HARD RULES (safety + quality rails — read first)

• Canvas (canvas.ust.hk): GET only. NEVER POST/PUT/PATCH/DELETE. NEVER start a quiz attempt. No exceptions you initiate. This is an always-enforce, catastrophic-if-missed rule — a mutation creates a graded submission record that cannot be un-created. Quizzes (incl. "Pop Quiz" / "Lecture Video" types) count for grade. To get quiz content, you do NOT touch the quiz — mine from slides / transcripts / past papers / GitHub mirrors instead.
  • There exists a practice-quiz pool-harvest technique that mutates state (it POSTs). You never run it and never decide it's safe. You cannot reliably tell graded-from-practice via a read, so you do not judge it — only the operator does. It runs ONLY when the operator, this instance, explicitly tells you to AND has confirmed the quiz is ungraded practice. Absent that exact instruction: treat it as forbidden and don't surface the endpoints.
• Don't surface-clone the sample/past paper. Concept overlap is fine; reusing its code/scenario stems is not.
• Don't ship a one-shot autonomous trivia file as primary material. Fully-autonomous self-graded subagents are LOW-TRUST (the 4021 trivia agent missed 4 of 7 worth-attempting traps). Build high-yield material interactively or with mid-flight review; diff against an answer key before declaring complete.
• Don't teach to the test when patching the cheatsheet — patch general-truth claims, not the user's specific blind spot.

The MCQ drill loop (the core)

One active 25-Q set at a time; everything else is source pool or graduated flashcards.

Initial spawn (once per course): read materials → group into modules → diagnostic gap probe → hand-author Set 1 + Set 2 as style anchors (don't delegate until tone is set) → dispatch parallel creator subagents for the depth-axis source banks → run verifier subagents (second pair of eyes — real, not theatre: caught wrong-answer-letter bugs) → concat all_notes.md for one-file Ctrl-F.

Refresh cycle (each round) — triage the user's attempt by reasoning quality, not just correctness (read the Thinking: line):

Then: patch the cheatsheet where a miss revealed a gap (the miss is the completeness signal) → build next set = revamps (you author inline — you know the misconception) + carryovers + pulls from source banks to refill to 25 → drain: delete pulled Qs from their source files (sources shrink monotonically) → verifier pass on freshly-authored revamps → update lineage/drain tables.

Headline metric: 🔁/🔄 repeat-test passes — miss in round N, pass the revamp in N+1 = concept internalized.

Why the loop is shaped this way (the learning science — and the one gap to close): the graduate/retest/revamp triage IS desirable difficulty — retrieval practice (recall from memory, not re-reading) + varying the shape. Name the trap it guards against: fluency strength (got it right now) ≠ storage strength (will retain it) — and getting a card right once feels like mastery it hasn't earned. Two mechanics the base loop under-uses, worth adding:

• Spacing: a graduated card that's gone forever optimizes fluency, not storage. Re-surface a fraction of graduated cards on a widening interval (round N+2, N+5…); a graduated card that misses on spaced re-exposure was fluency, not storage — send it back.
• Interleaving: don't drill one module to exhaustion then the next — mix modules within a set so retrieval has to first identify which rule applies (the real exam condition), not just apply a primed one. (Skills/procedure practice only; pure fact-recall doesn't need it.) Keep grounding every question in a primary source (slides/past-paper/key), never parametric recall — the HARD RULES already bind this. (Fold from mattpocock/skills teach, 2026-07-22.)

Set 0.x extrapolation track (after ≥2 rounds): group misses by cognitive failure shape (e.g. "mutation ≠ completion", "Pythonic-coercion leak", "adjacent-concept decision-tree gap", "API getter/setter asymmetry") and predict where the SAME shape will bite on un-drilled concepts. A Set 0.x miss = "the pattern hasn't generalized" = diagnostic gold. Out-of-syllabus is OK in moderation if flagged (≥70% in-syllabus, ≤30% flagged stretch).

Print-pack architecture

• Tag-freeze: after round-1 prints, git tag print-pack-v1-<date>. Never edit packed files in place (the printed copy is now the index of record); new material lands as to_print_md_roundN/ addenda with their own build_pack_roundN.py.
• Two-stage PDF for hard print-shop margins (Chrome --print-to-pdf silently shrinks-to-fit on overflow): Stage 1 pandoc+Chrome headless renders raw (@page{margin:0}, no margin-boxes); Stage 2 pypdf scale(0.93)-wraps each page onto a fresh letter (the scale envelope = inviolable ~0.30in white margin) + reportlab imprints headers in the outer band.
• Per-module N-up lectures: 9-up landscape, 3×3, slides rotated 90° CW; one PDF per module, not one book. Decide layout by visual A/B (render, look) and quantify utilization% first; confirm slide aspect (16:9 vs 4:3) before scaling.
• Pull-out maneuver: when the cheatsheet bloats with audience-specific content, pull detail to a dedicated primer and leave a one-line breadcrumb stub at the original spot ("references must be kept, else it's dangerous to read the other sheets with missing content").

Source mining

• Canvas (read-only — see HARD RULES): snapshot via the browser's own session — fetch(url,{credentials:'include'}) from a Canvas-origin Edge tab (HttpOnly cookie never leaves the browser). Works as a student: course/modules/assignments/files/folders/pages-by-slug/lecture-video file-IDs.
• Course site (e.g. course.cse.ust.hk/<code>/) when materials aren't on Canvas; check GitHub mirrors first (martin-mok/HKUST-COMP*) before any paywall.
• Past-paper syllabus-drift table before drilling any past paper: <year> topic | status in current syllabus — flag REMOVED topics (a 2020 paper had ~25% mass in removed topics).
• NotebookLM cram pods: 3 topic-coherent notebooks ≤20 sources each (quality degrades past ~20); Pod 1 = exam forensics (≤7: past papers + worked keys, "walk every question, don't pad"); tone = "TA the night before". 1-up PDFs only (verify pdftotext); markdown ingests natively.

Subagent dispatch + trust calibration

Trust ranking (highest 4021 post-mortem finding): interactive-build > vendor-sheet-with-annotation > autonomous-mined-trivia. Dispatch that works: exact paths + exact output shape + word budget + "OK | ISSUE: " format + hard tool-call cap (≤25). Doesn't work: answer-inference when curated answers exist, naive Read on >10 MB JSONL (Bash-grep first), "summarize" without a target schema. Author revamps yourself — you know the user's specific misconception; a subagent doesn't.

Course adaptation

When NOT to apply

• One-off exam not part of a series (tag-freeze/round-addenda is overkill).
• Closed-book exam (print-pack matters less; lean into drills + cram pods).
• User explicitly opts out of a step ("skip the diagnostic probe, time is short") — follow that.