Exam Ingest

exam-ingest — validated workspace initialization

Purpose

Convert a confirmed materials folder into a validated cram workspace. Build and repair the knowledge base only; do not teach or grade. The normal path produces structured ingestion facts under .ingest/, compiled chapter wiki and bank files, progress state, visual evidence, and an explicit readiness verdict before handing control back to exam-cram.

This module is the explicit processing_mode=full route. A missing, legacy, or lightweight processing choice must not activate it; route that learner through scripts/lightweight_session.py instead.

Activation

Activate when the confirmed workspace lacks its wiki, bank, or progress state; when the student supplies new/changed course materials; or when validate_workspace.py reports ingestion readiness blocked. Do not treat the mere existence of generated files as proof that the workspace is ready.

Inputs

• A student-confirmed materials directory containing PDF, DOCX, PPTX, XLSX, common standalone raster images, txt, or Markdown. Scans/images without usable sidecar text, damaged/encrypted files, unsupported features/formats, and ambiguous problem/solution pairs may require evidence-backed AI/human review.
• A target workspace directory explicitly confirmed by the student. Never default to the repository, process current directory, or an inferred course folder. The workspace must be separate from the materials tree so reruns cannot ingest generated outputs. If no workspace is confirmed, use update_progress.py workspace-list --json, then ask the student to select or provide one before writing anything.

Workflow

1. Pass the executable start gate, then use the official ingestion entry. The exact materials/workspace pair, all three learning choices, and explicit processing_mode=full must already have been persisted with exam_start.py confirm as specified by exam-cram; a bare registry row or update_progress.py set is insufficient. Verify read-only with exam_start.py status --materials <dir> --workspace <ws> --json; require ready_to_ingest=true, then run from the package root:

   python scripts/ingest_course.py --materials <dir> --workspace <ws> --json [--course-name <name>] [--lang zh|en] [--artifact-mode chat|visual]
   

   The default core orchestrator performs dependency preflight, deterministic extraction, provenance-preserving structured compilation, state initialization, visual indexing/repair, and canonical workspace validation. It never installs a dependency. Pass --artifact-mode only for an explicit standing student choice; omit it to retain the existing preference (or the default chat on a new workspace). An ordinary exam_start.py confirm with no --processing-mode likewise preserves an existing canonical processing choice; nevertheless this subskill still requires the effective choice to be explicit/current full.

2. Interpret process and readiness separately. Exit 0 means the engineering process completed and the JSON readiness is ready or usable_with_gaps; preserve and report any warnings in the latter. Exit 10 means process_success=true but readiness=blocked: do not teach, quiz, or claim completion. Any other nonzero is a dependency, input, or operation failure. For a missing required capability, ask once with the active language pack's consent line, install only on yes, then rerun the same command. A business/data failure is never evidence that Python is absent.

3. Require ingestion-v2 parser receipts. The regular path writes .ingest/parser_receipts.json with one receipt for every discovered source. Each row binds canonical source path, exact source SHA-256/media type, adapter/module/distribution/version, requested and produced location anchors, config SHA-256, result status, and the exact policy {network:false, upload:false, install:false}. Missing/duplicate rows, source or page drift, a policy mismatch, or a receipt referring to an unknown source blocks validation. A legacy ingestion-v1 payload remains readable only as legacy and must not be described as having v2 receipts. Unit language comes only from its payload: zxx is formula/symbol-only, never inherited, and never zh/en Guide support; otherwise review. Automatic layout crops remain available as unreceipted legacy crop_image assets for ordinary tutoring/quiz ingestion; geometry alone must never mint a current Study Guide receipt. Every new strict crop requires receipt schema v2 plus semantic-review schema v2, exact crop-hash binding, unrelated_content_present=false, and student_attempt_present=false. Target-only is the default (verdict=target_item_only, isolation=target_item_only, empty required_context_ids, detected IDs exactly the target); a dependent prompt instead uses verdict=target_with_required_context plus the distinct isolation=target_with_required_context, declares sorted unique prerequisite item/theorem/example IDs, and detects the target followed by exactly those contexts. Historical receipt schema v1 and semantic-review v1 stay read-only; existing v2 single-region receipts remain readable without hash/ID migration. For a completed ingestion-v2 workspace, scripts/backfill_crop_receipts.py validate|apply --workspace <ws> --annotations <jsonl> --json supports upgrade_existing, create_from_parent, and prompt-only create_composite_from_parent without rerunning the PDF builder. The composite is an explicit compatible v2 receipt variant: 2–32 non-overlapping regions from one exact parent/source/page are stacked without scaling using fully specified order, gap, RGBA background, and horizontal alignment; every pixel/PDF bbox, content ID, parent/target/candidate hash/dimension, source/parser revision, and output hash is bound and rechecked. It performs no OCR or arbitrary editing. A tainted parent page alone may not reject a semantically reviewed clean prompt component, but the candidate/output must be clean; every answer-side parent/target/candidate remains official-only. apply publishes only verified candidate bytes to a digest-named output plus raw/report/material-pending in a crash-recoverable locked transaction and invokes only the compiler; any failure stays fail-closed. See docs/crop-receipt-backfill.md. The normal orchestrator publishes .ingest/material_build_pending.json before any successful candidate asset/raw/report generation becomes visible. A nonzero builder result publishes none of those candidates, preserves the canonical raw input and parse report, and returns diagnostics only in the command result; if publication itself cannot roll back cleanly, the blocker is retained. Pending binds the prior build manifest, new raw/report, complete candidate asset policy, and exact migration receipt ledger. While it exists, ordinary ingestion publication/mutation—including review, claim, and Guide writers—and validation fail closed; only the explicit generation-aware builder/compiler path may proceed. Only a receipt-bijective answer_context -> student_attempt correction is migratable; standalone builder migration, stale bytes, missing/extra receipts, and every other role change fail closed. A pending generation plus a missing or drifted exam_runtime_receipt.json is recovered only through python scripts/exam_start.py recover-material-build --workspace <ws> --materials <dir> --action resume|supersede --json; ordinary confirm intentionally refuses. resume may compile only the exact pending generation: it skips parsing when both bound source documents are exact, permits blocker-first reconstruction when they are incomplete, and publishes nothing if reconstruction produces a different generation. Only an explicit supersede may publish that different candidate; its schema-2 pending marker binds the immediate predecessor. Audit records are generation-addressed under .ingest/material_build_recovery/, bounded to 64 authorization events and 64 direct predecessor edges. Every abandoned edge names its direct child. A final receipt binds at most those 64 rows plus one current completed resume row, and the build manifest hash-binds exactly that declared recovery-log set. Never remove or edit pending/recovery facts by hand. For that generation, the compiler places structured facts, the build manifest, wiki/bank/teaching layers, retrieval index, reports/plans, and the pending-to-receipt transition in one bounded ingestion transaction. It writes .ingest/pending_ingest.json and backups before the first registered target changes; validation blocks on a crash residue, and the next locked mutation restores all registered targets before continuing. Candidate assets/raw/report remain the builder generation outside this rollback set, so material pending stays available for an exact retry. Successful finalization writes .ingest/material_build_receipt.json, emits build-manifest schema 2 with an exact material_build contract and raw/report/receipt artifact hashes, re-verifies live bindings, and removes material pending last. Current-protocol output must not be refreshed or re-emitted as schema 1; legacy schema 1 remains readable but does not claim this gate. ingest_course.py performs later study_state.json initialization and optional artifact-preference writes only after compiler success; those learner-state operations are outside the compiler transaction. This protocol is lock-coordinated and crash-recoverable for process interruption, not a claim of power-loss durability or a filesystem-atomic snapshot for arbitrary unlocked readers.

4. Use the dedicated XLSX/raster routes and honest anchors. XLSX is parsed locally with the standard library: each worksheet is one page-equivalent and preserves workbook order, sparse cell coordinates/values, formulas plus stored cached values, defined-table metadata, and supported embedded raster assets without requiring Excel. The parser does not calculate formulas; missing cached values, external/network-looking formulas, hidden sheets, and unsupported relationships become typed review signals. A standalone raster is one page-equivalent with signature-checked dimensions/hash and a local source_page asset. Safe UTF-8 sidecars may supply text; otherwise emit standalone_raster_needs_ocr and route to an installed local OCR/vision capability or typed review—never fake empty-text success. PDF page values are page ordinals, PPTX values are slide ordinals, and DOCX values are logical segments split only at explicit page breaks; never call a DOCX anchor a physical rendered page.

5. Keep optional high-fidelity parsing explicit and remote. Never probe for, download, install, import, or execute a local Docling/MinerU package. Neither is an automatic fallback. Only after the learner explicitly requests the named parser may a host offer its own remote/cloud integration and separately disclose upload/privacy terms. The local CLI does not configure or call that service. A host must return revision-bound results through its remote boundary; otherwise report the integration unavailable and continue with core plus typed visual review. A local installation or callable local runner is never permission to use the heavy parser.

6. Check derived duplicate/conflict facts. In ingestion-v2, .ingest/duplicate_candidates.jsonl, canonical_groups.jsonl, source_conflicts.jsonl, and source_priorities.jsonl are deterministic derived facts, not mutable source truth. They bind exact content-unit/source revisions. Exact groups may choose a deterministic display occurrence while preserving every source occurrence and its location-derived unit_id; near matches are not folded automatically. Priority is evidence metadata, never an implicit winner. Any unresolved conflict fails closed and must be surfaced/resolved through evidence-backed review before teaching, quizzes, guide material claims, or completion.

7. Take over typed issues one by one. Treat .ingest/review_queue.jsonl as the canonical lifecycle, not .ingest/ai_review_manifest.json (legacy view only). A new type_defaulted issue is scoped to exactly one question/external ID; never close a source-wide legacy issue after checking only one chapter. For a gradable subjective question with an official paired answer but no grading points, subjective_keywords_missing targets the answer unit and binds the official answer source revision/pages. Add narrow source-backed metadata.keywords there; the compiler uses question-side keywords first and otherwise inherits the paired answer's reviewed keywords. No official answer means no inferred keywords. Start with:

   python scripts/ingest_review.py --workspace <ws> --json list
   python scripts/ingest_review.py --workspace <ws> --json show <issue_id>
   python scripts/ingest_review.py --workspace <ws> --json claim <issue_id>
   

   Read each issue's source hash, page/evidence references, reason codes, description, and suggested action. Recover scans/images through the host's available OCR/vision path; inspect ambiguous chapter or problem/solution assignments against the original pages; never infer an official answer from filename alone. A bare one-page Example N.M is likewise never an automatic answer. The builder leaves it as an unanswered teaching-only item and emits inline_worked_answer_candidate. After visual review has produced a current semantic-v2 full-prompt crop, use the explicit compiler-only route documented in docs/inline-worked-example-evidence.md: register-inline-worked with the exact existing question unit, unique same-page native material text unit, crop receipt, reviewer, and review note; then claim, draft-inline-worked, validate-patch, and apply. This route writes content-addressed evidence and the ordinary replace_unit + add_unit + pair_qa ledger patch. It does not rerun PDF parsing or mutate immutable source_raw_input. The answer must remain zh|en, same source revision/page/title/text, non-gradable, teaching-only, and bound by inline_material_source_unit_id to the exact native unit. Never place answer_origin=inline_material in quiz_bank. After an AI/human reviewer has finished a page-by-page visual audit of a batch of formula_hint issues, scripts/import_formula_audit.py --workspace <ws> --audit <audit.json> [--audit <more.json>] --output-dir <draft-dir> --reviewer <name> --json may convert that audit into deterministic evidence-bound patch drafts. It only drafts: it does not claim issues, apply patches, rebuild derivatives, or treat an audit-supplied render path as ledger evidence. Continue through ingest_review.py validate-patch and apply-batch --patch-list <draft-dir>/patch-list.json; the importer never replaces those gates.

8. Apply only evidence-bound patches. Build one strict ReviewPatch per issue from show and run validate-patch on every file. Use apply for one patch. For many inspected independent issues, apply-batch --patch-list <json> keeps separate context validation, transactions, and ledger identities while compiling derivatives once; partial progress remains replay-safe. Never combine issue identities. Allowed operations add/replace a unit, assign chapter/phase, pair Q&A, classify an asset, or mark unrecoverable. A cross-source pair_qa operation must include a sorted source_revisions binding for both the question and answer source revisions; drift on either side reopens review instead of replaying the old decision. Use mark-unrecoverable --reason ... only after recovery is impossible. Never hand-edit the append-only ledger, queue, compiled units, facts, wiki, or bank.

9. Rebuild and validate after review. Run ingest_review.py --workspace <ws> rebuild, then validate_workspace.py <ws> --json. Source drift, stale parser/fact hashes, unresolved conflicts or blocking issues, missing location anchors, or unbound blocking review entries keep readiness blocked. unrecoverable issues remain visible warnings rather than disappearing.

10. Account for every alert. Read the stable .ingest/parse_report.json, .ingest/unbound_review.json, typed queue, parser/fact warnings and conflicts, and ingest_report.json.missing_answer_ids in full. Recover each supported gap or tell the student exactly which material remains incomplete and why. Never silently skip an alert.

11. Advanced lower-level diagnostic path only. To isolate a compiler/parser defect, maintainers may run scripts/build_raw_input_from_workspace.py and then scripts/ingest.py directly. This is not the normal student workflow and does not replace final validation. Both workspace-publication commands recheck the exact confirmed pair, runtime receipt, complete learning choices, and processing_mode=full; direct invocation cannot bypass the lightweight boundary. scripts/ingest.py compiles a prepared payload; it does not independently prove readiness.

12. Three-sided visual cross-check AFTER ingest has created the workspace. The normal orchestrator already runs build_visual_index.py --apply --apply-wiki and recompiles. In lower-level diagnostics, inspect wiki visual coverage, prompt suspects, answer suspects, deferred answer pages, and shared prompt/answer blockers separately. A zero count on one side proves nothing about the others; answer-only pages never enter prompt/wiki context early.

13. True no-Python fallback only. Manual writing is allowed only after a direct interpreter probe proves Python truly cannot start. A nonzero command is a fail-loud operation error, not permission to degrade silently. In the confirmed fallback, disclose that structured validation, typed review, source-version/parser-receipt/conflict checks, and visual cross-checks are unavailable, then create only the minimum workspace from the selected locale templates. Missing package files are not evidence that Python is unavailable.

14. Label compiled provenance honestly: 🟢 来自资料 for material-derived content, 🟡 AI补充，可能与你老师讲的不完全一致 for an explicit supplement, and ⚠️ AI生成答案，非老师/教材提供 for a generated answer when no official answer exists.

Output Contract

• Return a readiness-aware receipt, not a generic success claim: ready may hand control to teaching; usable_with_gaps must name the warnings before teaching; blocked must state the issue count/reasons and remain in review.
• Produce .ingest/ structured facts including ingestion-v2 parser receipts and derived duplicate/group/conflict/priority sidecars, references/wiki/, references/quiz_bank.json, optional references/teaching_examples.json, append-only references/teaching_baseline.json, visual indices/assets, study_plan.md, study_state.json, generated study_progress.md, ingest_report.json, and a freshness-bound BM25 retrieval index.
• Every discovered source is recorded, and every location the selected adapter can enumerate is accounted for. Structured units retain source file/hash, location anchor, element kind, parent/section context, chapter/phase mapping, extraction method/confidence, and asset role where available. Blank/scanned known PDF pages still receive page anchors and review evidence; logical DOCX segments, PPTX slides, XLSX worksheets, and raster page-equivalents retain their honest anchor semantics. A file whose locations cannot be enumerated remains an explicit source-level review issue rather than disappearing.
• IDs are location identities, not content revisions: source_id derives from the canonical source path, and unit_id derives from source ID + page/bbox + kind + ordinal. Exact source and full-unit digests bind revisions and must accompany any dedup/conflict/claim evidence.
• Student-facing receipts use the persisted language: English by default, Simplified Chinese when the student opened in Chinese, or explicit bilingual composition. Machine JSON keys, hashes, IDs, reason codes, and statuses remain stable control-plane vocabulary.

Language packs

Load the matching student wording before emitting a receipt:

• 中文 → ../../locales/zh/skills/exam-ingest.md
• English → ../../locales/en/skills/exam-ingest.md
• 双语 → compose both packs block by block, Chinese first with a > EN: mirror

zh, en, and bilingual are the persisted canonical values. 中文, English, and 双语 are display/legacy input aliases normalized before storage.

Boundaries

• The package-root scripts and locale templates are required. If this subskill is installed alone, report the packaging error and use/install the complete package. Missing package files are not evidence that Python is unavailable and do not authorize manual fallback.
• Do not modify parser/compiler logic while acting as the exam coach. Use the public commands and typed patch lifecycle.
• Do not fabricate a standard answer, source filename, page, chapter assignment, or review resolution.
• Do not hand control back to teaching while validator readiness is blocked.
• Ingestion establishes location/revision and derived conflict facts; it does not invent authored Guide claims. In ingestion-v2, exam-study-guide later binds material assertions to exact same-unit source refs, writes the location-only claim receipt, and lets the typed-guide validator recompute that gate.
• Keep scripts/retrieve.py BM25 as the default. Dense, RRF, and reranker helpers are experimental only and cannot enter the student path until a sufficient frozen real multi-course recall Gold Set passes the documented optional-backend gate; the committed synthetic sample is explicitly insufficient evidence.
• Only an explicitly requested remote/cloud host may implement the LangGraph contract; local graph construction is disabled. Remote graph checkpoints and resume values are never workspace truth. Re-read current state, .ingest/, runtime, guide, and QA receipts at each guarded transition.