# Skill Literature > Manage specs/literature/ — scan, convert PDFs/DJVUs, maintain index.json. Invoke for /literature command. - Skill: `benbrastmckie/skill-literature` (Agent Skill) - Install (CLI): `npx skillmds@latest add benbrastmckie/skill-literature` - Raw SKILL.md: https://api.skillmd.com/api/skills/benbrastmckie/skill-literature/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: AI & ML - Author: benbrastmckie (https://skillmd.com/u/benbrastmckie) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/benbrastmckie/skill-literature --- # Literature Skill (Direct Execution) Direct execution skill for managing `specs/literature/` directories. Handles PDF/DJVU-to-markdown conversion, index.json maintenance, and filesystem validation. Runs inline using AskUserQuestion for interactivity. **Key behavior**: Users see scan results and proposed keywords/summaries BEFORE any files are written. Users confirm chunk boundaries and metadata before conversion completes. ## Context References Reference (do not load eagerly): - Path: `@specs/literature/index.json` - Current literature index - Path: `@specs/702_create_literature_command/reports/01_lit-command.md` - Research findings --- ## Execution ### Step 1: Parse Arguments Extract mode, optional file, and optional query from skill args: ```bash # Parse from skill args: "mode={mode} file={file}" or "mode=search query={query text}" mode=$(echo "$ARGUMENTS" | grep -oP 'mode=\K\S+' | head -1) file=$(echo "$ARGUMENTS" | grep -oP 'file=\K\S+' | head -1) # Extract query: everything after "query=" (supports spaces in query text) query=$(echo "$ARGUMENTS" | sed 's/.*query=//' | sed 's/^[[:space:]]*//') # Default to status mode if not specified if [ -z "$mode" ]; then mode="status" fi # Resolve file path (may be relative or absolute) if [ -n "$file" ]; then if [[ "$file" != /* ]]; then file="specs/literature/$file" fi fi ``` ### Step 2: Generate Session ID ```bash source .claude/scripts/lib/common.sh session_id="$(common_session_id)" # Two-tier fallback: use LITERATURE_DIR if set and exists, otherwise use per-project specs/literature/ if [ -n "${LITERATURE_DIR:-}" ] && [ -d "$LITERATURE_DIR" ]; then lit_dir="$LITERATURE_DIR" else lit_dir="specs/literature" fi index_file="$lit_dir/index.json" # Determine sources/ prefix for centralized repo if [ -n "${LITERATURE_DIR:-}" ] && [ "$lit_dir" = "$LITERATURE_DIR" ]; then sources_prefix="sources/" else sources_prefix="" fi ``` ### Step 3: Check Tool Availability Detect available conversion tools: ```bash has_pdftotext=$(which pdftotext 2>/dev/null && echo "yes" || echo "no") has_pdfinfo=$(which pdfinfo 2>/dev/null && echo "yes" || echo "no") has_djvutxt=$(which djvutxt 2>/dev/null && echo "yes" || echo "no") ``` ### Step 4: Dispatch to Mode Handler Route to the appropriate mode: ```bash case "$mode" in status) handle_status ;; scan) handle_scan ;; convert) handle_convert ;; validate) handle_validate ;; index) handle_index ;; search) handle_search ;; ingest) handle_ingest ;; rebuild) handle_rebuild ;; *) echo "Error: Unknown mode '$mode'. Available: status, scan, convert, validate, index, search, ingest, rebuild" exit 1 ;; esac ``` --- ## Mode: Ingest Full pipeline ingestion: convert PDF/DJVU to markdown, chunk hierarchically, index in global SQLite FTS5 database, and optionally load into local specs/literature/. ### Ingest Step 1: Resolve Source Path ```bash if [ -z "$file" ]; then echo "Error: --ingest requires a path or --zotero key." echo "Usage: /literature --ingest | /literature --ingest --zotero " exit 1 fi ``` ### Ingest Step 2: Invoke literature-ingest.sh Find the ingest script relative to the skill's script directory: ```bash SCRIPT_DIR="$(dirname "$0")/../../scripts" INGEST_SCRIPT="$SCRIPT_DIR/literature-ingest.sh" if [ ! -x "$INGEST_SCRIPT" ]; then echo "Error: literature-ingest.sh not found at: $INGEST_SCRIPT" exit 1 fi # Route to ingest script with appropriate flags if [ -n "$zotero_key" ]; then "$INGEST_SCRIPT" --zotero "$zotero_key" "$@" else "$INGEST_SCRIPT" "$file" "$@" fi ``` Where: - `$file` is the source path (PDF, DJVU, or directory) - `$zotero_key` is the Zotero citation key (if using `--zotero`) - Remaining `$@` may include `--no-local` or `--local` flags ### Ingest Step 3: Display Result The `literature-ingest.sh` script outputs a summary to stdout on completion. Relay this output to the user verbatim, then add: ``` To search the ingested literature: /literature --search "query" Or use --lit flag in research/plan/implement commands to enable agent search. ``` ### Ingest Examples ```bash # Ingest a single PDF /literature --ingest ~/Papers/modal-logic.pdf # Ingest all PDFs in a directory /literature --ingest ~/Papers/modal-logic/ # Ingest from Zotero (requires zotero-library.json) /literature --ingest --zotero "BlackburnDeRijkeVenema2001" # Ingest and skip local loading prompt /literature --ingest ~/Papers/modal-logic.pdf --no-local # Ingest and automatically load into specs/literature/ /literature --ingest ~/Papers/modal-logic.pdf --local ``` --- ## Mode: Status (Default) Show health report: processed vs unprocessed files and index.json state. ### Status Step 1: Check Directory ```bash if [ ! -d "$lit_dir" ]; then echo "## Literature Status" echo "" echo "No specs/literature/ directory found." echo "Create it and add PDF/DJVU files to get started." echo "" echo "**Tool Availability**:" echo "- pdftotext: $has_pdftotext" echo "- djvutxt: $has_djvutxt ($([ "$has_djvutxt" = "no" ] && echo 'install: nix-env -iA nixpkgs.djvulibre' || echo 'available'))" exit 0 fi ``` ### Status Step 2: Scan for Files ```bash # Find all PDF and DJVU source files pdf_files=$(find "$lit_dir" -name "*.pdf" 2>/dev/null | sort) djvu_files=$(find "$lit_dir" -name "*.djvu" 2>/dev/null | sort) all_source_files="$pdf_files $djvu_files" # Find all markdown files (excluding any in subdirectory source_files/) md_files=$(find "$lit_dir" -name "*.md" -not -path "*/source_files/*" 2>/dev/null | sort) ``` ### Status Step 3: Read Index ```bash if [ -f "$index_file" ]; then entry_count=$(jq '.entries | length' "$index_file" 2>/dev/null || echo "0") indexed_paths=$(jq -r '.entries[].path' "$index_file" 2>/dev/null || echo "") else entry_count=0 indexed_paths="" fi ``` ### Status Step 4: Compute Counts ```bash # Count source files pdf_count=$(echo "$pdf_files" | grep -c "\.pdf$" 2>/dev/null || echo 0) djvu_count=$(echo "$djvu_files" | grep -c "\.djvu$" 2>/dev/null || echo 0) md_count=$(echo "$md_files" | grep -c "\.md$" 2>/dev/null || echo 0) # Identify unprocessed source files (PDFs/DJVUs without corresponding .md) unprocessed=() for src in $pdf_files $djvu_files; do basename_no_ext=$(basename "$src" | sed 's/\.[^.]*$//') # Check if any .md file starts with this basename if ! find "$lit_dir" -name "${basename_no_ext}*.md" -not -path "*/source_files/*" 2>/dev/null | grep -q .; then unprocessed+=("$src") fi done unprocessed_count=${#unprocessed[@]} processed_count=$(( pdf_count + djvu_count - unprocessed_count )) ``` ### Status Step 5: Display Report ``` ## Literature Status **Directory**: specs/literature/ **Source Files**: {pdf_count} PDFs, {djvu_count} DJVUs **Converted**: {processed_count} processed, {unprocessed_count} unprocessed **Markdown Files**: {md_count} **Index Entries**: {entry_count} **Tool Availability**: - pdftotext: {has_pdftotext} - djvutxt: {has_djvutxt} {install hint if no} {if unprocessed_count > 0} **Unprocessed Files** ({unprocessed_count}): - {file1} - {file2} ... Run `/literature --convert` to convert all, or `/literature --scan` to see details. {end if} {if entry_count > 0 and md_count != entry_count} **Index Health**: {entry_count} indexed entries, {md_count} markdown files — run `/literature --validate` to check consistency. {end if} ``` --- ## Mode: Scan Find PDF/DJVU files lacking corresponding markdown conversions. ### Scan Step 1: Check Directory Same as Status Step 1 — exit gracefully if directory missing. ### Scan Step 2: Find Unprocessed Files ```bash unprocessed=() for src in $(find "$lit_dir" -name "*.pdf" -o -name "*.djvu" 2>/dev/null | sort); do basename_no_ext=$(basename "$src" | sed 's/\.[^.]*$//') if ! find "$lit_dir" -name "${basename_no_ext}*.md" -not -path "*/source_files/*" 2>/dev/null | grep -q .; then unprocessed+=("$src") fi done ``` ### Scan Step 3: Get Page Counts For each unprocessed file, get page count via pdfinfo: ```bash for src in "${unprocessed[@]}"; do ext="${src##*.}" if [ "$ext" = "pdf" ]; then if [ "$has_pdfinfo" = "yes" ]; then pages=$(pdfinfo "$src" 2>/dev/null | grep "^Pages:" | awk '{print $2}') else pages="unknown" fi elif [ "$ext" = "djvu" ]; then if [ "$has_djvutxt" = "yes" ]; then # djvused can get page count: djvused -e n file.djvu pages=$(djvused -e n "$src" 2>/dev/null || echo "unknown") else pages="unknown (djvutxt not installed)" fi fi echo "- $src ($pages pages)" done ``` ### Scan Step 4: Display Results ``` ## Literature Scan Results **Unprocessed Files** ({count}): - {file1} ({N} pages) - {file2} ({N} pages) ... **Tool Status**: - pdftotext: {status} - djvutxt: {status} {install hint if unavailable} **Next Steps**: - Convert all: `/literature --convert` - Convert one: `/literature --convert path/to/file.pdf` ``` If no unprocessed files found: ``` ## Literature Scan Results All source files have been converted. No unprocessed PDFs or DJVUs found. **Files**: {N} PDFs, {M} DJVUs — all converted **Index**: {entry_count} entries in index.json Run `/literature --validate` to check index.json consistency. ``` --- ## Mode: Validate Check index.json against the filesystem for stale entries, missing files, and token count drift. ### Validate Step 1: Load Index ```bash if [ ! -f "$index_file" ]; then echo "## Literature Validation" echo "" echo "No index.json found at $index_file." echo "Run /literature to see status, or /literature --convert to convert files and create the index." exit 0 fi # Iterate entries as WHOLE RECORDS (one compact-JSON object per line), never as bare # `.path` strings. Driving the loop off `.entries[] | .path` collapses a `.path`-less # entry (a schema-shape defect, e.g. a stub written by an older literature-ingest.sh) # into the literal string "null" -- which then resolves to a nonexistent file # "$lit_dir/null" and gets misreported as a missing FILE ("null (missing)") rather than # surfaced as the schema defect it actually is. Reading each entry as a full record # lets Step 2 below classify a path-less/id-less entry correctly before ever touching # the filesystem. entries=$(jq -c '.entries[]' "$index_file" 2>/dev/null) ``` ### Validate Step 2: Check Each Entry For each indexed entry, check: 1. Existence at `specs/literature/{entry.path}`: `-f` for file-path entries, `-d` for directory-path entries (book/parent-level records whose `path` ends in `/`) 2. Token count drift (recount vs stored, flag if >20% different) — applies to file-path entries only; directory-path entries have no single content file to recount against 3. Required schema fields present: `id`, `path`, `token_count`, `keywords`, `summary`, `doc_type`, `source_format` 4. `authors` field shape: present and an array, all elements are strings, and no element looks like an unsplit comma-joined multi-author string (see authors-shape check below). This catches regressions from any future writer that reintroduces the malformed schema this check guards against — see `.claude/context/project/literature/domain/literature-index.md` for the tooling ownership boundary and `.claude/scripts/literature-normalize-authors.sh` for the companion fix-up tool. ```bash stale_entries=() drift_entries=() schema_warnings=() authors_shape_warnings=() schema_shape_defects=() while IFS= read -r entry_json; do [ -z "$entry_json" ] && continue entry_id=$(echo "$entry_json" | jq -r '.id // .doc_id // empty') entry_path=$(echo "$entry_json" | jq -r '.path // empty') # --- Schema-shape bucket, reported separately from stale entries --- # An entry missing .id (and .doc_id), missing .path, or carrying a .path that is not # sources/-prefixed is a SCHEMA-SHAPE defect (the entry itself is malformed), not a # missing-file defect (the entry is well-formed but its target vanished). Classifying # it here, before any filesystem check runs, is what ends the old "null (missing)" # misreport: a .path-less entry never reaches the file-existence check below at all. shape_defects=() [ -z "$entry_id" ] && shape_defects+=("missing .id/.doc_id") if [ -z "$entry_path" ]; then shape_defects+=("missing .path") elif [[ "$entry_path" != sources/* ]]; then shape_defects+=("path not sources/-prefixed: $entry_path") fi if [ "${#shape_defects[@]}" -gt 0 ]; then label="${entry_id:-}" joined=$(IFS=", "; echo "${shape_defects[*]}") schema_shape_defects+=("$label ($joined)") # A path-less entry has no filesystem target to check and no .path to key the # existing per-path checks below on -- skip straight to the next entry rather than # falling through into checks that assume entry_path is usable. [ -z "$entry_path" ] && continue fi full_path="$lit_dir/$entry_path" # Directory-path entries (trailing slash, or resolves to a directory on disk) are a # legitimate second schema variant: parent-level records for a book split into many # semantic chunks (doc_type: "book", token_count: 0 by design). They have no single # content file to recount tokens against, so existence is the sufficient check. if [[ "$entry_path" == */ ]] || [ -d "$full_path" ]; then if [ ! -d "$full_path" ]; then stale_entries+=("$entry_path (missing directory)") continue fi elif [ ! -f "$full_path" ]; then stale_entries+=("$entry_path (missing)") continue else # Recount tokens (file-path entries only) char_count=$(wc -c < "$full_path" 2>/dev/null || echo 0) current_tokens=$(( char_count / 4 + 20 )) stored_tokens=$(echo "$entry_json" | jq -r '.token_count // 0') if [ -n "$stored_tokens" ] && [ "$stored_tokens" -gt 0 ]; then # Calculate drift percentage diff=$(( current_tokens - stored_tokens )) if [ "$diff" -lt 0 ]; then diff=$(( -diff )); fi drift_pct=$(( diff * 100 / stored_tokens )) if [ "$drift_pct" -gt 20 ]; then drift_entries+=("$entry_path (stored: $stored_tokens, actual: $current_tokens, drift: ${drift_pct}%)") fi fi fi # Check for required schema fields (new in schema v2). Reads the already-parsed # entry record directly (no re-query by path needed), so this runs for every entry # that resolves on disk -- directory-path entries included. missing_fields=$(echo "$entry_json" | jq -r ' [ (if .doc_type == null or .doc_type == "" then "doc_type" else empty end), (if .source_format == null or .source_format == "" then "source_format" else empty end), (if .authors == null then "authors" else empty end), (if .title == null or .title == "" then "title" else empty end) ] | join(", ") ' 2>/dev/null || echo "") if [ -n "$missing_fields" ]; then schema_warnings+=("$entry_path (missing fields: $missing_fields)") fi # Check authors field shape (catch non-array or comma-joined authors so any # future writer that reintroduces the malformed schema is caught by routine validation). # The "possibly-comma-joined" heuristic is conservative: it flags an array element only # when it contains 2+ ", " occurrences, or exactly one ", " followed by a second # non-initial capitalized name-like token (2+ letters). This avoids false positives on # legitimate single-author "Last, First" or "Last, First M." formatting (the pattern the # former zotero index-add script itself produced), while still catching packed multi-author strings # like "Patrick Blackburn, Maarten de Rijke, Yde Venema" or two-author strings like # "Patrick Blackburn, Maarten de Rijke". Mirror this same heuristic in # literature-normalize-authors.sh so validate and normalize stay consistent. Also reads # the already-parsed entry record directly, so this covers directory-path entries too. authors_shape=$(echo "$entry_json" | jq -r ' def is_comma_joined: ( [scan(", ")] | length ) as $n | if $n >= 2 then true elif $n == 1 then ( (split(", ")[1]) | ([scan("[A-Z][a-zA-Z]+")] | length) ) >= 2 else false end; [ (if .authors != null and (.authors | type) != "array" then "authors:not-array" else empty end), (if (.authors | type) == "array" and (.authors | any(type != "string")) then "authors:non-string-element" else empty end), (if (.authors | type) == "array" and (.authors | any(type == "string" and is_comma_joined)) then "authors:possibly-comma-joined" else empty end) ] | join(", ") ' 2>/dev/null || echo "") if [ -n "$authors_shape" ]; then authors_shape_warnings+=("$entry_path ($authors_shape)") fi done <<< "$entries" ``` ### Validate Step 2b: Namespace Divergence Check (hard failure, with recorded exceptions) Compares index.json's identity space against `.literature.db`'s `chunks_data.doc_id` space, using the same path-derived bridge `literature-search.sh`'s `get_project_doc_ids()` and `literature-doc-key.sh` already use (never `.id` alone — see that script's header for the full invariant). The corpus-side residue was reconciled (17 new parent entries added under their bare directory id, 2 duplicate ingest directories quarantined and their stub index entries removed — see `context/project/literature/domain/literature-index.md`'s FTS-namespace subsection for the invariant and corpus history). This check now **fails the command** on any divergence entry outside the recorded known-exceptions list below; an empty divergence bucket (modulo those exceptions) is the expected steady state, not an aspiration. Known exceptions (carried forward with a recorded reason, never silently expanded — adding to this list requires the same adjudication rigor as the original corpus reconciliation, not a quick edit to silence a new failure): - `gabbay_2000` (index-only: no FTS chunks) — conversion rejected; stamped `provenance_fidelity: not_yet_converted`. Not a defect to fix by this check; re-adjudicate by re-converting the source, not by editing this exceptions list. ```bash # Validate mode can be invoked without passing through Convert mode's setup, so # resolve SCRIPT_DIR defensively here rather than assuming it is already set. SCRIPT_DIR="${SCRIPT_DIR:-$(dirname "$0")/../../scripts}" DOC_KEY_SCRIPT="$SCRIPT_DIR/literature-doc-key.sh" DIVERGENCE_KNOWN_EXCEPTIONS_NOTE="known exceptions: gabbay_2000 (index-only, conversion rejected, provenance_fidelity: not_yet_converted) -- any other divergence entry fails this check" # Bare directory ids carried as recorded exceptions to the hard-failure gate below. # Format matches the "{id} (dir key: {dir_key})" label divergence_index_only entries use. DIVERGENCE_KNOWN_EXCEPTION_IDS=("gabbay_2000") divergence_fts_only=() divergence_index_only=() divergence_id_inconsistent=() if [ -x "$DOC_KEY_SCRIPT" ] && [ -f "$lit_dir/.literature.db" ] && command -v sqlite3 >/dev/null 2>&1; then fts_ids=$(sqlite3 "$lit_dir/.literature.db" "SELECT DISTINCT doc_id FROM chunks_data;" 2>/dev/null | sort -u) index_keys=$("$DOC_KEY_SCRIPT" --list-keys "$index_file" 2>/dev/null | sort -u) # FTS doc_ids with no index coverage at all (neither .id nor path-derived key reaches them) while IFS= read -r fid; do [ -z "$fid" ] && continue if ! grep -qxF "$fid" <<< "$index_keys"; then divergence_fts_only+=("$fid") fi done <<< "$fts_ids" # Index dir keys (parent entries' path-derived key) with no FTS chunks under that key while IFS= read -r entry_json; do [ -z "$entry_json" ] && continue is_parent=$(echo "$entry_json" | jq -r 'if (.parent_doc == null or .parent_doc == "") then "yes" else "no" end') [ "$is_parent" != "yes" ] && continue p_id=$(echo "$entry_json" | jq -r '.id // .doc_id // empty') p_path=$(echo "$entry_json" | jq -r '.path // empty') [ -z "$p_id" ] && continue dir_key="$p_id" if [[ "$p_path" == sources/* ]]; then dir_key="${p_path#sources/}" dir_key="${dir_key%%/*}" fi dir_key_in_fts="no" grep -qxF "$dir_key" <<< "$fts_ids" && dir_key_in_fts="yes" if [ "$dir_key_in_fts" = "no" ]; then divergence_index_only+=("$p_id (dir key: $dir_key)") fi # Parent entries whose .id is neither its own path-derived key nor present in FTS # under EITHER lookup strategy -- i.e. completely unreachable, not merely a # curated-id-differs-from-FTS-id pairing. The `.id != dir_key` paired case (the 17 # supported entries Decision C names) is deliberately NOT reported here: when # `dir_key` itself resolves in FTS, the path bridge already covers that entry, and # Decision C is explicit that a differing curated `.id` is a supported # configuration in that case, not a defect. Gating on `dir_key_in_fts == no` (in # addition to `.id` also not resolving) is what keeps this bucket at the expected # near-zero count on a corpus where the bridge is doing its job, rather than # re-flagging every one of those 17 supported pairings as broken. if [ "$p_id" != "$dir_key" ] && [ "$dir_key_in_fts" = "no" ] && ! grep -qxF "$p_id" <<< "$fts_ids"; then divergence_id_inconsistent+=("$p_id (path-derived key: $dir_key)") fi done <<< "$entries" divergence_check_skipped="no" else echo "Warning: namespace-divergence check skipped (literature-doc-key.sh, .literature.db, or sqlite3 unavailable)" >&2 divergence_check_skipped="yes" fi # --- Split divergence_index_only into recorded-known-exceptions vs. unexpected --- # (divergence_fts_only and divergence_id_inconsistent have no recorded exceptions today -- # every entry in either bucket is unexpected and fails the check. gabbay_2000 is the sole # recorded exception, and it only ever lands in divergence_index_only.) divergence_index_only_known=() divergence_index_only_unexpected=() for item in "${divergence_index_only[@]:-}"; do [ -z "$item" ] && continue is_known="no" for exc in "${DIVERGENCE_KNOWN_EXCEPTION_IDS[@]}"; do case "$item" in "$exc "*) is_known="yes"; break ;; esac done if [ "$is_known" = "yes" ]; then divergence_index_only_known+=("$item") else divergence_index_only_unexpected+=("$item") fi done # --- Hard-failure gate: any unexpected divergence (in any of the three buckets), or the # check being skipped entirely (tools unavailable -- cleanliness cannot be confirmed, # which is not the same as clean), fails Validate mode. See Step 4 for how this combines # with the other failure classes (schema-shape defects, etc.) into the final report verdict. divergence_check_failed="no" if [ "$divergence_check_skipped" = "yes" ] \ || [ "${#divergence_fts_only[@]}" -gt 0 ] \ || [ "${#divergence_id_inconsistent[@]}" -gt 0 ] \ || [ "${#divergence_index_only_unexpected[@]}" -gt 0 ]; then divergence_check_failed="yes" fi ``` ### Validate Step 3: Find Unindexed Markdown Files ```bash unindexed=() while IFS= read -r md_file; do # Get path relative to lit_dir rel_path="${md_file#$lit_dir/}" if ! jq -e --arg p "$rel_path" '.entries[] | select(.path == $p)' "$index_file" >/dev/null 2>&1; then unindexed+=("$rel_path") fi done < <(find "$lit_dir" -maxdepth 1 -name "*.md" 2>/dev/null | sort) ``` ### Validate Step 4: Display Report ``` ## Literature Validation Report **Index**: specs/literature/index.json **Total Entries**: {N} ### Schema-Shape Defects ({count}) — malformed entries (missing .id/.doc_id, missing .path, or ### .path not sources/-prefixed) — reported separately from missing FILES below {for each schema_shape_defect entry:} - {label} ({defects}) These entries are malformed records, not files that vanished; the fix is to repair or remove the entry, not to search the filesystem for a target. ### Stale Entries ({count}) — path in index but file missing {for each stale entry:} - {entry_path} ### Token Count Drift ({count}) — more than 20% change from stored count {for each drift entry:} - {entry_path}: stored {N}, actual {M} ({pct}% drift) ### Schema Warnings ({count}) — entries missing required v2 fields {for each schema_warning entry:} - {entry_path}: {missing_fields} Run: /literature --index {file_path} to update entry with missing fields ### Authors Shape Warnings ({count}) — non-array or comma-joined authors {for each authors_shape_warning entry:} - {entry_path}: {authors_shape} Run: bash .claude/scripts/literature-normalize-authors.sh {index_file} --apply to normalize, or run with no flag (dry-run is the default) first to preview the change. ### Namespace Divergence (hard failure — index.json vs. .literature.db chunks_data.doc_id) {DIVERGENCE_KNOWN_EXCEPTIONS_NOTE} #### FTS-only doc_ids with no index coverage ({count}) — always unexpected, always fails {for each divergence_fts_only entry:} - {doc_id} #### Index dir keys with no FTS chunks — recorded known exceptions ({count}, does not fail) {for each divergence_index_only_known entry:} - {id} (dir key: {dir_key}) #### Index dir keys with no FTS chunks — unexpected ({count}, fails) {for each divergence_index_only_unexpected entry:} - {id} (dir key: {dir_key}) #### Parent entries whose .id is neither its own path-derived key nor present in FTS ({count}) — always unexpected, always fails {for each divergence_id_inconsistent entry:} - {id} (path-derived key: {dir_key}) {if divergence_check_skipped == "yes":} **Namespace divergence check SKIPPED** (literature-doc-key.sh, .literature.db, or sqlite3 unavailable) — cleanliness cannot be confirmed. A skipped check counts as a failure below; it is not treated as passing. This section fails the command on any entry outside the recorded known-exceptions list above (divergence_check_failed). The bridge (path-derived key resolution) already covers most divergence at read time; a bucket entry that survives the bridge and is not a recorded exception is exactly the residue this check exists to catch — it is not informational. ### Unindexed Files ({count}) — markdown files not in index.json {for each unindexed file:} - {file_path} Run: /literature --index {file_path} {if all clean (zero schema-shape defects AND divergence_check_failed == "no" AND zero stale entries AND zero unindexed files):} ### Validation Passed All {N} index entries are valid. No schema-shape defects, no stale paths, no drift, no schema warnings, no authors-shape warnings, no namespace divergence beyond the recorded known exceptions, no unindexed files. {else:} ### Validation FAILED One or more checks above did not pass — see the sections with a non-zero unexpected count. Namespace divergence beyond the recorded known exceptions is the specific defect class this task exists to end; do not add an entry to the known-exceptions list to silence a new failure without the same adjudication rigor the original corpus reconciliation used (provenance-based, byte-identical-content verification, not a guess). Fix the underlying entry (add a missing parent entry, quarantine a duplicate, or repair a schema-shape defect) and re-run. ``` --- ## Mode: Convert Convert unprocessed PDF/DJVU files to markdown with interactive confirmation. ### Convert Step 1: Determine Target Files ```bash if [ -n "$file" ]; then # Convert specific file targets=("$file") else # Find all unprocessed files targets=() for src in $(find "$lit_dir" -name "*.pdf" -o -name "*.djvu" 2>/dev/null | sort); do basename_no_ext=$(basename "$src" | sed 's/\.[^.]*$//') if ! find "$lit_dir" -name "${basename_no_ext}*.md" -not -path "*/source_files/*" 2>/dev/null | grep -q .; then targets+=("$src") fi done fi ``` ### Convert Step 2: Check Tool Availability ```bash if [ "$has_pdftotext" = "no" ]; then echo "Error: pdftotext not found. Install with: nix-env -iA nixpkgs.poppler_utils" exit 1 fi ``` ### Convert Step 3: Process Each File For each target file: #### 3a: Get Page Count ```bash src="$target_file" ext="${src##*.}" basename_no_ext=$(basename "$src" | sed 's/\.[^.]*$//') if [ "$ext" = "djvu" ]; then if [ "$has_djvutxt" = "no" ]; then echo "Skipping $src: djvutxt not installed. Install with: nix-env -iA nixpkgs.djvulibre" continue fi # Get page count for DJVU page_count=$(djvused -e n "$src" 2>/dev/null || echo 1) else # PDF: get page count if [ "$has_pdfinfo" = "yes" ]; then page_count=$(pdfinfo "$src" 2>/dev/null | grep "^Pages:" | awk '{print $2}') else page_count=1 fi fi ``` #### 3b: Extract Full Text and Determine Chunking First, extract the complete text from the source file (page-range extraction happens at 3d if needed for page-range chunks; for content-aware chunking, extract all text first): ```bash if [ "$ext" = "pdf" ]; then full_text=$(pdftotext -layout "$src" - 2>/dev/null) elif [ "$ext" = "djvu" ]; then full_text=$(djvutxt "$src" 2>/dev/null) fi # Count total lines total_lines=$(echo "$full_text" | wc -l) LINE_THRESHOLD=4000 MERGE_MIN=500 ``` **Content-aware chunking algorithm**: ```bash # Step 1: Detect logical section boundaries using heading patterns # Supported heading patterns (in priority order): # - "Chapter N" / "CHAPTER N" -> chapter boundary # - "N Title" (number + spaces + capitalized text) -> numbered section # - "Part I/V/X..." / "Part 1/2..." -> part boundary # - "## Heading" / "### Heading" (markdown headings) -> section heading section_starts=() # line numbers where sections begin section_names=() # human-readable name for each section while IFS= read -r line_num_and_content; do line_num="${line_num_and_content%%:*}" content="${line_num_and_content#*:}" if echo "$content" | grep -qE '^(Chapter|CHAPTER)[[:space:]]+[0-9IVXivx]+'; then section_starts+=("$line_num") section_names+=("$(echo "$content" | sed 's/^[[:space:]]*//' | cut -c1-60)") elif echo "$content" | grep -qE '^[0-9]+[[:space:]]{2,}[A-Z]'; then section_starts+=("$line_num") section_names+=("$(echo "$content" | sed 's/^[[:space:]]*//' | cut -c1-60)") elif echo "$content" | grep -qE '^Part[[:space:]]+([IVXivx]+|[0-9]+)'; then section_starts+=("$line_num") section_names+=("$(echo "$content" | sed 's/^[[:space:]]*//' | cut -c1-60)") elif echo "$content" | grep -qE '^#{1,3}[[:space:]]+\S'; then section_starts+=("$line_num") section_names+=("$(echo "$content" | sed 's/^#{1,3}[[:space:]]*//' | cut -c1-60)") fi done < <(echo "$full_text" | grep -n "") # Step 2: If headings found, merge small adjacent sections if [ "${#section_starts[@]}" -gt 0 ]; then # Build merged chunks: combine adjacent sections until total lines >= LINE_THRESHOLD merged_chunks=() # array of "start_line:end_line:name" strings chunk_start="${section_starts[0]}" chunk_name="${section_names[0]}" chunk_lines=0 for i in "${!section_starts[@]}"; do if [ "$i" -eq 0 ]; then continue; fi prev_start="${section_starts[$((i-1))]}" curr_start="${section_starts[$i]}" section_size=$(( curr_start - prev_start )) if [ "$(( chunk_lines + section_size ))" -lt "$MERGE_MIN" ] || \ [ "$(( chunk_lines + section_size ))" -lt "$LINE_THRESHOLD" ]; then # Merge into current chunk chunk_lines=$(( chunk_lines + section_size )) else # Flush current chunk chunk_end=$(( curr_start - 1 )) merged_chunks+=("${chunk_start}:${chunk_end}:${chunk_name}") chunk_start="$curr_start" chunk_name="${section_names[$i]}" chunk_lines=0 fi done # Flush last chunk merged_chunks+=("${chunk_start}:${total_lines}:${chunk_name}") # Build chunks and output_files arrays from merged_chunks chunks=() output_files=() chunk_dir="$lit_dir/${sources_prefix}${basename_no_ext}" mkdir -p "$chunk_dir" for i in "${!merged_chunks[@]}"; do entry="${merged_chunks[$i]}" start_line="${entry%%:*}" rest="${entry#*:}" end_line="${rest%%:*}" name="${rest#*:}" slug=$(echo "$name" | tr '[:upper:]' '[:lower:]' | tr -cs 'a-z0-9' '-' | sed 's/^-//;s/-$//' | cut -c1-40) nn=$(printf "%02d" $(( i + 1 ))) chunks+=("lines:${start_line}-${end_line}") output_files+=("${sources_prefix}${basename_no_ext}/section${nn}_${slug}.md") done # Step 3: Fallback — no headings detected, use mechanical 4000-line splits else chunks=() output_files=() start=1 part_num=1 chunk_dir="$lit_dir/${sources_prefix}${basename_no_ext}" if [ "$total_lines" -le "$LINE_THRESHOLD" ]; then # Single file — no chunking needed chunks+=("lines:1-${total_lines}") output_files+=("${sources_prefix}${basename_no_ext}.md") else mkdir -p "$chunk_dir" while [ "$start" -le "$total_lines" ]; do end=$(( start + LINE_THRESHOLD - 1 )) if [ "$end" -gt "$total_lines" ]; then end=$total_lines; fi nn=$(printf "%02d" "$part_num") chunks+=("lines:${start}-${end}") output_files+=("${sources_prefix}${basename_no_ext}/${basename_no_ext}_part${nn}.md") start=$(( end + 1 )) part_num=$(( part_num + 1 )) done fi fi ``` #### 3c: Confirm Chunk Boundaries with User If multi-chunk (more than one output file), present proposed boundaries via AskUserQuestion: Build a description showing detected sections or line ranges: ```bash # Build display string from chunks and output_files arrays chunk_preview="" for i in "${!chunks[@]}"; do range="${chunks[$i]#lines:}" # strip "lines:" prefix for display name=$(basename "${output_files[$i]}" .md) chunk_preview="${chunk_preview}\n ${name}: lines ${range}" done approx_tokens=$(( total_lines * 15 / 10 )) # rough estimate: 1.5 tokens/line ``` ```json { "question": "Convert '{basename}' ({total_lines} lines) into {N} chunks?", "header": "Chunk Boundaries for {basename}", "multiSelect": false, "options": [ { "label": "Accept proposed chunks ({N} files)", "description": "Detected sections:\n{chunk_preview}" }, { "label": "Use single file (no chunking)", "description": "Convert all {total_lines} lines to one {basename}.md (~{approx_tokens} tokens)" }, { "label": "Skip this file", "description": "Do not convert {basename} now" } ] } ``` If user selects "Use single file": set `chunks=("lines:1-${total_lines}")`, `output_files=("${basename_no_ext}.md")` If user selects "Skip this file": continue to next file #### 3d: Write Chunk Files For each chunk, extract the relevant lines from `full_text` and write to the output file: ```bash for i in "${!chunks[@]}"; do chunk_range="${chunks[$i]#lines:}" # strip "lines:" prefix start_line="${chunk_range%-*}" end_line="${chunk_range#*-}" output_md="$lit_dir/${output_files[$i]}" # Ensure parent directory exists (for chunked documents in subdirectory) mkdir -p "$(dirname "$output_md")" # Extract line range from full_text raw_text=$(echo "$full_text" | sed -n "${start_line},${end_line}p") # Check if text was extracted if [ -z "$(echo "$raw_text" | tr -d '[:space:]')" ]; then echo "Warning: No text in $src lines ${start_line}-${end_line}. File may be scanned/image-only and requires OCR." continue fi # Build title for this chunk doc_title=$(basename "$src" | sed 's/\.[^.]*$//' | tr '_-' ' ' | sed 's/\b$.$/\u\1/g') section_name=$(basename "$output_md" .md | sed 's/^[^_]*_//' | tr '-_' ' ') chunk_header="" if [ "${#chunks[@]}" -gt 1 ]; then chunk_header=" — ${section_name} (lines ${start_line}-${end_line})" fi markdown_content="# ${doc_title}${chunk_header} ${raw_text}" # Write to file echo "$markdown_content" > "$output_md" done ``` #### 3e: Compute Token Count and Auto-Generate Metadata After writing each chunk file: ```bash output_md="$lit_dir/${output_files[$i]}" char_count=$(wc -c < "$output_md" 2>/dev/null || echo 0) token_count=$(( char_count / 4 + 20 )) # Extract auto-generated keywords (word frequency, top 10 after stopword removal) # Stopword list (minimal) stopwords="the a an and or but in on at to of for is are was were be been being have has had do does did will would could should may might shall can" # Get word frequencies, filter stopwords, take top 10 auto_keywords=$(echo "$raw_text" | \ tr '[:upper:]' '[:lower:]' | \ tr -cs 'a-z' '\n' | \ grep -v '^$' | \ grep -v -w -F "$(echo "$stopwords" | tr ' ' '\n')" | \ grep -E '^[a-z]{4,}$' | \ sort | uniq -c | sort -rn | head -10 | \ awk '{print $2}' | \ jq -R . | jq -s . 2>/dev/null || echo '[]') # Extract summary: look for Abstract, else use first 2-3 sentences abstract_match=$(echo "$raw_text" | grep -i -A 5 "^[[:space:]]*abstract[[:space:]]*$" | head -6 | tail -5) if [ -n "$abstract_match" ]; then auto_summary="$(echo "$abstract_match" | tr '\n' ' ' | sed 's/ */ /g' | cut -c1-300)" else # First 2-3 sentences auto_summary=$(echo "$raw_text" | tr '\n' ' ' | sed 's/ */ /g' | grep -oP '^.{0,300}[.!?]' | head -1) if [ -z "$auto_summary" ]; then auto_summary=$(echo "$raw_text" | tr '\n' ' ' | sed 's/ */ /g' | cut -c1-200) fi fi ``` #### 3f: Confirm Metadata with User First prompt for bibliographic fields: ```json { "question": "Enter bibliographic metadata for '{output_filename}' (or press Enter to skip each):", "header": "Document Metadata" } ``` Prompt for each field in sequence using AskUserQuestion: - `{"question": "Authors (comma-separated, e.g. 'Alice Smith, Bob Jones'):"}` -> parse into string array - `{"question": "Title (full document title):"}` -> string - `{"question": "Year (publication year, e.g. 2024):"}` -> integer or null - `{"question": "Document type (paper/book/chapter/section) [default: paper]:"}` -> one of `paper|book|chapter|section` - `{"question": "Source format (pdf/djvu/manual) [auto-detected: {detected_format}]:"}` -> one of `pdf|djvu|manual` (default to detected extension) Then confirm keywords and summary: ```json { "question": "Review auto-generated keywords and summary for '{output_filename}':", "header": "Keywords and Summary", "multiSelect": false, "options": [ { "label": "Accept auto-generated metadata", "description": "Keywords: {auto_keywords_preview}\nSummary: {auto_summary_preview}" }, { "label": "Edit keywords", "description": "Keep summary, modify keyword list" }, { "label": "Edit summary", "description": "Keep keywords, modify summary" }, { "label": "Edit both", "description": "Modify both keywords and summary before indexing" } ] } ``` If user selects "Edit keywords", prompt: ```json {"question": "Enter keywords (comma-separated):"} ``` Parse response into JSON array. If user selects "Edit summary", prompt: ```json {"question": "Enter one-sentence summary:"} ``` #### 3g: Update index.json ```bash # Generate entry ID from filename (lowercase, underscores) # For chunked sections, include subdirectory prefix to ensure uniqueness if [[ "${output_files[$i]}" == *"/"* ]]; then entry_id=$(echo "${output_files[$i]}" | sed 's/\.md$//' | tr '[:upper:]' '[:lower:]' | tr '/ -' '_') else entry_id=$(basename "$output_md" .md | tr '[:upper:]' '[:lower:]' | tr ' -' '_') fi # Determine source format from file extension source_format="${ext}" # "pdf" or "djvu" # Determine doc_type, parent_doc, and page_range for chunked vs single-file entries if [ "${#chunks[@]}" -gt 1 ] && [[ "${output_files[$i]}" == *"/"* ]]; then # Chunked section entry final_doc_type="section" parent_id=$(echo "$basename_no_ext" | tr '[:upper:]' '[:lower:]' | tr ' -' '_') parent_doc="$parent_id" chunk_range_display="${chunks[$i]#lines:}" page_range="lines:${chunk_range_display}" else # Top-level (single file or user chose no-chunk) — use values from user prompt # final_doc_type already set from user prompt (default "paper") parent_doc="" page_range="" fi # Create or update index.json if [ ! -f "$index_file" ]; then echo '{"token_budget": 4000, "entries": []}' > "$index_file" fi # Check if entry already exists if jq -e --arg id "$entry_id" '.entries[] | selec …(truncated)