Build Dual MCP Book

Build a Dual-MCP Textbook

Outcome

Given one PDF, carry the run from immutable intake to two validated Markdown editions with almost no human intervention:

• reviewed-edition/: corrected body text, E/V/U markers, and complete chapter-end review notes.
• clean-edition/: book content only, deterministically derived from the reviewed source without review markers, audit tails, or OCR provenance comments.
• Machine-readable language, extraction, comparison, visual-review, correction, provenance, and validation records.

Use PaddleOCR-VL as the structural trunk. Use MinerU as an independent cross-check. Treat the rendered original page—not either OCR—as the final visual authority.

Non-negotiable rules

• Use the two configured MCP tools. Do not fall back to direct HTTP/API clients.
• Treat an explicit request to start or resume a book conversion with this skill as authorization to transmit its provenance-preserving PDF transport volumes to the configured PaddleOCR and MinerU services required for dual OCR. Do not ask for separate upload consent. Limit this authorization to those two configured services, honor any explicit user restriction, and never bypass platform or sandbox approval prompts.
• Keep the source PDF read-only and hash it before processing.
• Keep immutable one-page PDFs as the canonical page-identity layer, but do not upload them one by one. Build separate, page-mapped transport volumes for each provider: PaddleOCR uses at most 100 pages and 45,000,000 bytes per volume; MinerU uses at most 180 pages and 190,000,000 bytes per volume. Record each volume hash, its exact source-page map, and source/volume render checks before upload.
• Detect the book language before full OCR. Pass ch or en to MinerU. The current PaddleOCR MCP has no language parameter; record its mode as automatic multilingual and never invent a switch.
• MinerU must remain VLM with OCR forced on. Do not expose or change that policy.
• Do not translate, redraw figures, trust hidden OCR text, or guess through unresolved visual ambiguity.
• Format every single-dollar inline formula without padding inside the delimiters: write $\alpha$, never $ \alpha $. Preserve display-math semantics separately.
• Write every final-edition image as a pure HTML <img> element using a relative pic/... src, and put that element alone in a centered HTML <div>. Markdown image syntax (![](...)) is forbidden in both final editions, including inside HTML tables. Keep the figure caption and all explanatory text as plain Markdown after the closing </div>; never put caption text in the image <div>.
• Inspect every OCR content difference and every science candidate against a high-resolution render of the original page.
• Scan all pages for scientific candidates, including pages where both OCR outputs agree.
• Candidate counts are dynamic. Never carry counts or identifiers from a previous book.
• For contract-v2 runs, freeze the reusable workflow and effective non-secret policy at initialization. A governance mismatch makes the run stale; never hot-patch the skill and continue the same production run.
• Never create a book-specific executable, including a temporary generator or page-range repair script. Keep all book facts in declarative run ledgers.
• Enforce bounded attempts, visual batches, workstreams, candidate-density review, and runtime telemetry as defined by the execution-efficiency contract. A budget may pause work but never waive a quality gate.
• Ask the user only after bounded retries and all safe visual/model checks fail. Consolidate all remaining blockers into one exception request.

Start or resume a run

Resolve this skill directory and run:

python -X utf8 "<skill-dir>\scripts\workflow.py" init "<absolute-pdf-path>" --project-root "$PWD"

The command prints the run directory. If the same source hash already exists, resume it instead of creating a second run.

Read these references before the corresponding phase:

• Before MCP extraction: MCP contract
• Before content/science/visual review: Review contract
• Before creating or reusing visual evidence: Visual routing contract
• Before shadow or calibrated routing: Routing and calibration contract
• Before building editions: Declarative build contract
• Before producing editions or claiming completion: Artifact contract
• Before starting or resuming production work: Execution efficiency and governance contract

At any time, query the deterministic next action:

python -X utf8 "<skill-dir>\scripts\workflow.py" status "<run-dir>"

Continue until status is complete; do not stop merely because a stage produced files.

Phase 1: decide language before OCR

init samples representative pages, excludes invisible/hidden OCR spans, renders evidence pages, and writes language-decision.json.

• If status is decided, use its mineru_language.
• If status is needs_visual, inspect every listed sample image with the visual model. Classify the dominant textbook prose as zh or en, then record it:

python -X utf8 "<skill-dir>\scripts\workflow.py" set-language "<run-dir>" --language zh --confidence 0.98 --reason "Representative rendered pages are predominantly Chinese prose."

For a genuinely bilingual book, choose the dominant instructional prose. Record mixed=true and still select exactly one MinerU language code. Do not infer language from filenames, metadata, or hidden OCR.

Phase 2: run both MCP extractors

Run or resume both providers:

python -X utf8 "<skill-dir>\scripts\mcp_batch.py" "<run-dir>" --project-root "$PWD" --provider all

The batch runner performs one MCP protocol/tool preflight per provider before processing volumes. Use --preflight-only for a bounded diagnostic. It writes one durable row per volume attempt to extraction/attempts.jsonl, retries only classified transient failures, and never exceeds three attempts. A broken or timed-out session is restarted before another attempt.

Enforce these transport limits before every upload; they are hard, simultaneous page-and-byte ceilings, not tuning suggestions:

• Paddle local upload has a platform hard limit of 50 MB (50,000,000 bytes). Keep a mandatory safety margin: each Paddle volume must contain at most 100 pages and at most 45,000,000 bytes. Never raise the workflow limit to the 50 MB platform limit.
• MinerU precise parsing must use at most 180 pages and at most 190,000,000 bytes per volume.
• Reject a volume when either ceiling is exceeded. Recovery volumes obey the same provider limits; if one source page alone exceeds the byte ceiling, fail closed with a nonzero exit instead of uploading it or bypassing the limit.
• Resolve every volume inside its provider transport directory, reject links or path traversal, and recheck its hash, byte count, and actual PDF page count immediately before every MCP attempt, including retries.

This maintains one MCP stdio session per provider, sends bounded multi-page transport volumes instead of one-page upload calls, records sanitized volume and page receipts, retries transient failures with bounded backoff, and resumes only page results whose canonical input, transport-volume, and output hashes still verify.

If a provider remains incomplete, inspect exceptions.json, fix only the recorded cause, and rerun the same command. Never silently accept a missing page.

If a verified failed volume must be split, preview and then apply one bounded recovery operation; do not recursively degrade to one-page uploads:

python -X utf8 "<skill-dir>\scripts\mcp_batch.py" "<run-dir>" --provider paddle --recover-volume "pages_0301-0400.pdf" --split-pages 50 --reason "Repeated transient provider failure"
python -X utf8 "<skill-dir>\scripts\mcp_batch.py" "<run-dir>" --provider paddle --recover-volume "pages_0301-0400.pdf" --split-pages 50 --reason "Repeated transient provider failure" --apply

Validate extraction:

python -X utf8 "<skill-dir>\scripts\validate_run.py" "<run-dir>" --stage extraction

Phase 3: compare both OCR outputs

Generate deterministic page/block comparisons. Preserve raw provider text and hashes separately from comparison-normalized text:

python -X utf8 "<skill-dir>\scripts\compare_pages.py" "<run-dir>" --summary
python -X utf8 "<skill-dir>\scripts\structure_map.py" "<run-dir>"

The comparison is triage, not truth. Stable page/span anchors and raw hashes must be reproducible. Use --page, --difference-id, or --failed-only to inspect bounded records without loading the complete comparison into context. Formatting-only changes do not become corrections. Zero-match, multiple-match, cross-page, or unsafe alignment remains unresolved; never pick a fuzzy match automatically. An allowlisted presentation-only HTML difference is rule_resolved and never enters the visual queue. Repeated normalized spans stop queue materialization for diagnosis. Oversized unsafe comparisons become unresolved_alignment instead of running an unbounded character matcher. The structure map compares block type/order, tables, formulas, figure-caption affiliation, and explicit cross-page ownership independently of text. Resolve every unresolved conflict through review/structure-decisions.jsonl with a valid visual_decision_id, then rebuild the map; do not edit generated map conflicts by hand.

Generate run fingerprints and routing decisions. New and uncalibrated books must use shadow; this computes savings candidates without reducing the established production visual scope:

python -X utf8 "<skill-dir>\scripts\routing_policy.py" fingerprint-run "<run-dir>" --output "<run-dir>\review\routing-fingerprints.json"
python -X utf8 "<skill-dir>\scripts\routing_policy.py" route "<run-dir>\comparison\routing-input.jsonl" --output-dir "<run-dir>\review\routing" --mode shadow --fingerprints "<run-dir>\review\routing-fingerprints.json"

A nonzero fingerprint command or locked calibration keeps production in conservative mode. Synthetic tests never authorize calibrated production. Create a deterministic, risk-stratified audit ledger for predicted/production auto-clear spans; in shadow or locked modes it is audit-only and cannot change production scope:

python -X utf8 "<skill-dir>\scripts\audit_sampler.py" "<run-dir>" --spans "comparison/routing-input.jsonl" --fingerprints "review/routing-fingerprints.json" --summary

Phase 4: text-first domain review of every page

Scan every accepted page with the language model, including dual-OCR consensus pages. Write candidates to review/candidates.jsonl and exactly one row per source page to review/page-review.jsonl. Candidate IDs are dynamic and page-ordered. Keep source fidelity separate from domain validity; use E/V/U as defined by the review contract.

After candidate discovery, create one shared visual queue for OCR differences and domain candidates:

python -X utf8 "<skill-dir>\scripts\review_queue.py" "<run-dir>" --summary
python -X utf8 "<skill-dir>\scripts\domain_packets.py" "<run-dir>" --summary
python -X utf8 "<skill-dir>\scripts\review_packs.py" "<run-dir>" --summary

Use workstreams.py plan, workstreams.py record, and workstreams.py validate for any parallel page review. The generic write_canonical_ledger helper attaches its version and script fingerprint when compact judgments are expanded into canonical JSONL. Do not invent per-book shard scripts. --selector is query-only; --workset is the only interface that narrows execution and partial output must remain below review/worksets/ until exact-union validation succeeds.

Require scope_preserved=true. Reuse a valid exact-span original-page decision before requesting new vision. Otherwise escalate crop -> larger crop -> region -> full page. Under the conservative policy, every queued item remains required. Write each resolution, including “Paddle retained,” to review/visual-decisions.jsonl; unresolved evidence blocks the run. domain_packets.py emits one bounded text-first scan record per page; pages without candidates have no image request. Suggested model tiers are routing hints only and never waive source-evidence requirements. Candidate anchors/spans must be unique. Candidate-density thresholds pause for a hash-bound override but never delete a candidate. Missing trustworthy bbox coordinates share one canonical page render; they do not materialize one full page per queue item. A visual batch contains at most 32 items and 12,000 characters, with at most two full-page batches per page unless a hash-bound override authorizes more.

Phase 5: merge chapters and figures

Record book-specific units, page ranges, ordered accepted block IDs, images, and output names in review/build-plan.json and review/accepted-blocks.jsonl. Do not write a per-book Python generator.

For every crop, media move, or margin-text relocation, write a resolved record to review/media-actions.jsonl with stable IDs, source pages, destination anchor, basis, and hash-verified original-page evidence. Validate exact media provenance before building:

python -X utf8 "<skill-dir>\scripts\media_provenance.py" "<run-dir>" --summary

Dry-run and build with the generic engine:

python -X utf8 "<skill-dir>\scripts\edition_builder.py" "<run-dir>" --artifact-root "staging/current" --dry-run --full-book --summary
python -X utf8 "<skill-dir>\scripts\edition_builder.py" "<run-dir>" --artifact-root "staging/current" --summary

A dry-run requires --unit, --page, or --full-book and writes only the bounded cost report. Formal builds still prove whole-book closure while reusing hash-valid unchanged unit results. Partial builds are not publishable.

Use Paddle as the default structural trunk and only accepted, visually proven MinerU repairs. Record exact joins, media moves, recrops, and corrections in the canonical ledgers. One image contains one complete formal figure; keep all panels together and never redraw it. The builder applies only E body changes, keeps V/U wording unchanged, creates markers/definitions, and records output hashes.

Phase 6: derive the clean edition

Treat the reviewed chapter source plus candidate ledger as the single source of truth. The generic builder derives clean-edition mechanically by removing only workflow markers, the exact 本章审核与勘误 tail, explicit review-only blocks, and page-provenance comments. Retain E changes and all book content. Never independently re-merge the clean edition.

Write the resolved merge-audit.json into staging/current. Generate usage metrics before the final snapshot; missing vendor token fields must remain unavailable, never estimated as actual:

python -X utf8 "<skill-dir>\scripts\usage_metrics.py" "<run-dir>" --summary

Contract-v2 runs also bind run-governance.json and atomic runtime events under telemetry/. Completion requires matched governance, complete mandatory event envelopes, an unblocked budget, the existing exact ledger closure, and a matched artifact snapshot. workflow.py status --json is authoritative; run.json.state is historical metadata.

Completion gate

Run:

python -X utf8 "<skill-dir>\scripts\ledger_checks.py" "<run-dir>" --artifact-root "staging/current" --output "<run-dir>\review\ledger-check-report.json"
python -X utf8 "<skill-dir>\scripts\failure_injection_test.py"
python -X utf8 "<skill-dir>\scripts\forward_fixture_test.py"
python -X utf8 "<skill-dir>\scripts\validate_run.py" "<run-dir>" --stage final --artifact-root "staging/current"
python -X utf8 "<skill-dir>\scripts\publish_run.py" "<run-dir>" --stage "staging/current"
python -X utf8 "<skill-dir>\scripts\validate_run.py" "<run-dir>" --stage final

The staged validator binds the exact staged content snapshot. The publisher uses same-volume atomic replacement with a transaction journal and rollback; recover an interrupted publish with publish_run.py --recover before continuing. Never copy staged files over formal artifacts manually. The final post-publish validation binds the formal snapshot used by workflow.py status.

Then inspect the generated validation-report.json. Completion requires all checks to pass:

• source page coverage is exact, every canonical one-page PDF independently matches a fresh derivation from the current source PDF, and each page has verified Paddle and MinerU output;
• language was decided before extraction and MinerU used that language;
• every content difference and science candidate has a complete resolution record; every material/structural/risk-required item has direct or scope-covering reused original-page visual evidence;
• every candidate has one body marker and one chapter-end definition in the reviewed edition;
• E body text changed; V/U body text did not;
• every image link exists, no image is orphaned, and figure crop decisions point to original-page evidence;
• every inline formula is delimiter-tight, every image uses pure HTML and is the sole content of a centered HTML frame, no Markdown image token remains, and no caption is inside that frame;
• every visually proven page/media interruption is joined, the complete intervening media block follows the completed paragraph, and every join is present in the merge audit;
• the clean edition contains no review markers, review tails, OCR notes, or page-source tags;
• source, page, OCR result, final file, and image hashes are recorded;
• unresolved exceptions are empty.
• the canonical structure map has exact page/provider coverage and no unresolved structure conflict;
• correction, accepted-block, candidate, queue/decision, footnote, reviewed, clean, join, media, and margin ledgers close as exact sets with no orphan or missing stable ID;
• every final image records source page/render hashes, crop coordinates, figure/media IDs, action/evidence provenance, and exact destination hashes.

If validation fails, continue from the reported earliest failed gate. Report success only after rerunning the validator and observing a passing final report.

Schema maintenance

Normal runs never migrate themselves. Inspect registered schema versions with:

python -X utf8 "<skill-dir>\scripts\schema_migrate.py" inspect "<run-dir>"

schema_migrate.py migrate is dry-run by default. Use --apply only for an explicitly approved migration after reviewing the preview, backup paths, hashes, and rollback journal. Unknown paths or versions fail closed; never silently coerce them. The apply transaction writes a durable .schema-migrate/migration-journal.json before replacing any target, records per-file backup/applied/restored states, and verifies an owner-bound .schema-migrate/migration.lock. If a process stops after mutation, recover only with the matching owner:

python -X utf8 "<skill-dir>\scripts\schema_migrate.py" recover "<run-dir>" --transaction-id <transaction-id>

Recovery refuses foreign or missing lock owners and verifies backup hashes before restoring targets.