Doc To Markdown

Reading and Converting Documents

Instructions

1. Decide whether to convert at all

Conversion is a cost, not a default. Read the file directly when the question is bounded and the runtime can open the format — direct reading preserves figures, equations, and layout that no text conversion keeps, and lets you work a page range at a time. Where the runtime cannot open PDFs directly, convert, but convert only the pages you need.

Read directly when the document is under ~20 pages and the task is one pass over it (summarize, answer a question, check a claim, review a draft), when the payload is visual (figures, plots, scanned tables, slides), or when you need a page number you can cite.

Convert when the document is longer than one read comfortably spans, when the text will be searched or quoted repeatedly, when it joins a corpus or source library, when many documents get the same treatment, or when a downstream tool (grep, a classifier, a script) needs plain text. $citation-check and $fact-check both require converted Markdown, one file per source.

State which path you took in one line. Silently converting a two-page memo wastes a step; silently reading page 1 of a 300-page book and answering as if you read it all is worse.

2. Route by what the document actually is, not by its extension

Probe first — the extension says nothing about structure:

pdfinfo doc.pdf | grep -E '^(Pages|Encrypted)'
pages=$(pdfinfo doc.pdf | awk '/^Pages:/{print $2}')
chars=$(pdftotext -q doc.pdf - | wc -c)
echo "$((chars / pages)) chars/page"   # < 300 means image-only, needs OCR

Fall back to pandoc for .docx only when Node is unavailable; it escapes apostrophes, hard-wraps lines, and emits Pandoc-flavored rather than GitHub-flavored Markdown.

If the repo already ships a conversion script — scripts/convert-sources.sh and a sources/og + sources/md pair is the house layout — run that script instead of a bare converter. It encodes the project's naming and output conventions, and bypassing it produces files the rest of the pipeline cannot find.

3. Know each converter's failure mode

These are measured on the library's own corpus of political-science PDFs, not vendor claims:

• opendataloader-pdf leaves typographic ligatures intact. One 30-page article carried 426 raw fi/ff/ffi characters, so grep significant silently misses every hit. Normalize every conversion: python3 -c "import sys,unicodedata; sys.stdout.write(unicodedata.normalize('NFKC', open(sys.argv[1]).read()))" in.md > out.md. Verify with grep -c 'fi\|fl\|ff\|ffi' — the answer must be 0.
• opendataloader-pdf fails silently on scanned PDFs. A 28-page image-only article produced 2.5 KB of noise and exit code 0. Always run the chars-per-page probe first; never trust a short output.
• opendataloader-pdf under-detects section headings on some layouts, emitting the title as the only #. Check the heading count before relying on the Markdown for structure-aware chunking.
• anydoc invents tables out of two-column prose and footnote blocks, shredding sentences into pipe cells. On the same article it emitted 24 table rows where the PDF has no tables at all. Grep the output for ^| and read what it caught before trusting it on a prose document.
• anydoc merges a whole page into one line, running headers and footnotes into the middle of body paragraphs — fine for retrieval, wrong for quotation and for anything that reads paragraph structure.
• anydoc errors loudly on image-only PDFs (OCR is required, exit 1). That is a feature: use it as a cheap scanned-document detector.

After any conversion, spot-check the head, the tail, and one middle page against the original. Converters truncate, drop columns, and reorder without complaining.

4. Decide whether the Markdown persists or is thrown away

Make this call yourself, then say which way you went and where the file is.

Persist — write into the project and keep it — when any of these hold:

• The repo has a source-library convention (sources/md/, knowledge_base/md/, a convert-sources.sh). Write there, under the repo's naming scheme (author-year-slug.md, up to three authors, then firstauthor-etal-year-slug).
• The document is a cited source for a manuscript, or will be quoted, coded, or fact-checked.
• The document will be read again — a corpus, a literature review, a multi-session project.
• Conversion was expensive: OCR, a VLM run, a long book, a manual repair pass.

Use a scratch file and delete it when done when any of these hold:

• The document lives outside the project (Downloads, an email attachment, /tmp) and the task is one question about it.
• Conversion is cheap and exactly reproducible from a file that is not going anywhere.
• The request is transient — "what does this say about X", "check this number".
• The document is third-party copyrighted material and the repo does not already track converted sources. Repos that keep originals gitignored and conversions tracked have made that call deliberately; a repo with no such convention has not.

When in doubt, persist inside the project and tell the user, since a stray Markdown file is cheap and a re-run of a 40-minute OCR job is not. Two rules hold either way: never write into a git-tracked directory without saying so, and never delete or move the source document.

5. Record what produced the file

A converted document is derived data, and a reader six months on cannot tell a clean extraction from a mangled one. For anything persisted, note the converter and version — in the file's front matter, in sources/README.md, or in the conversion log the repo already keeps. When a document needed OCR, that fact travels with it; downstream text analysis has to know it is working from OCR output.

Bulk intake of many sources at once is $research-repo's job, not this skill's. Cleaning OCR output is $post-ocr-cleanup's.

Quality Checks

• Read-versus-convert decided explicitly and stated, not defaulted to conversion
• Structure probed before routing: page count and chars-per-page checked, image-only documents caught before a text extractor ran
• Repo's own conversion script used when the project ships one
• Ligatures normalized: grep -c 'fi\|fl\|ff\|ffi' returns 0
• Output spot-checked at head, tail, and one middle page against the original
• Tables inspected if the output contains any, and confirmed to correspond to tables in the source
• Persist-or-discard decided, stated, and acted on — scratch files actually deleted
• Naming convention followed for anything written into a source library
• Provenance recorded for persisted conversions, including whether OCR was involved
• Source document untouched