Skill Literature

Literature Skill (Direct Execution)

Direct execution skill for managing specs/literature/ directories. Handles PDF/DJVU-to-markdown conversion, index.json maintenance, and filesystem validation. Runs inline using AskUserQuestion for interactivity.

Key behavior: Users see scan results and proposed keywords/summaries BEFORE any files are written. Users confirm chunk boundaries and metadata before conversion completes.

Context References

Reference (do not load eagerly):

• Path: @specs/literature/index.json - Current literature index
• Path: @specs/702_create_literature_command/reports/01_lit-command.md - Research findings

Execution

Step 1: Parse Arguments

Extract mode, optional file, and optional query from skill args:

# Parse from skill args: "mode={mode} file={file}" or "mode=search query={query text}"
mode=$(echo "$ARGUMENTS" | grep -oP 'mode=\K\S+' | head -1)
file=$(echo "$ARGUMENTS" | grep -oP 'file=\K\S+' | head -1)

# Extract query: everything after "query=" (supports spaces in query text)
query=$(echo "$ARGUMENTS" | sed 's/.*query=//' | sed 's/^[[:space:]]*//')

# Default to status mode if not specified
if [ -z "$mode" ]; then
  mode="status"
fi

# Resolve file path (may be relative or absolute)
if [ -n "$file" ]; then
  if [[ "$file" != /* ]]; then
    file="specs/literature/$file"
  fi
fi

Step 2: Generate Session ID

source .claude/scripts/lib/common.sh
session_id="$(common_session_id)"
# Two-tier fallback: use LITERATURE_DIR if set and exists, otherwise use per-project specs/literature/
if [ -n "${LITERATURE_DIR:-}" ] && [ -d "$LITERATURE_DIR" ]; then
  lit_dir="$LITERATURE_DIR"
else
  lit_dir="specs/literature"
fi
index_file="$lit_dir/index.json"
# Determine sources/ prefix for centralized repo
if [ -n "${LITERATURE_DIR:-}" ] && [ "$lit_dir" = "$LITERATURE_DIR" ]; then
  sources_prefix="sources/"
else
  sources_prefix=""
fi

Step 3: Check Tool Availability

Detect available conversion tools:

has_pdftotext=$(which pdftotext 2>/dev/null && echo "yes" || echo "no")
has_pdfinfo=$(which pdfinfo 2>/dev/null && echo "yes" || echo "no")
has_djvutxt=$(which djvutxt 2>/dev/null && echo "yes" || echo "no")

Step 4: Dispatch to Mode Handler

Route to the appropriate mode:

case "$mode" in
  status)   handle_status ;;
  scan)     handle_scan ;;
  convert)  handle_convert ;;
  validate) handle_validate ;;
  index)    handle_index ;;
  search)   handle_search ;;
  ingest)   handle_ingest ;;
  rebuild)  handle_rebuild ;;
  *)
    echo "Error: Unknown mode '$mode'. Available: status, scan, convert, validate, index, search, ingest, rebuild"
    exit 1
    ;;
esac

Mode: Ingest

Full pipeline ingestion: convert PDF/DJVU to markdown, chunk hierarchically, index in global SQLite FTS5 database, and optionally load into local specs/literature/.

Ingest Step 1: Resolve Source Path

if [ -z "$file" ]; then
  echo "Error: --ingest requires a path or --zotero key."
  echo "Usage: /literature --ingest <path> | /literature --ingest --zotero <key>"
  exit 1
fi

Ingest Step 2: Invoke literature-ingest.sh

Find the ingest script relative to the skill's script directory:

SCRIPT_DIR="$(dirname "$0")/../../scripts"
INGEST_SCRIPT="$SCRIPT_DIR/literature-ingest.sh"

if [ ! -x "$INGEST_SCRIPT" ]; then
  echo "Error: literature-ingest.sh not found at: $INGEST_SCRIPT"
  exit 1
fi

# Route to ingest script with appropriate flags
if [ -n "$zotero_key" ]; then
  "$INGEST_SCRIPT" --zotero "$zotero_key" "$@"
else
  "$INGEST_SCRIPT" "$file" "$@"
fi

Where:

• $file is the source path (PDF, DJVU, or directory)
• $zotero_key is the Zotero citation key (if using --zotero)
• Remaining $@ may include --no-local or --local flags

Ingest Step 3: Display Result

The literature-ingest.sh script outputs a summary to stdout on completion. Relay this output to the user verbatim, then add:

To search the ingested literature: /literature --search "query"
Or use --lit flag in research/plan/implement commands to enable agent search.

Ingest Examples

# Ingest a single PDF
/literature --ingest ~/Papers/modal-logic.pdf

# Ingest all PDFs in a directory
/literature --ingest ~/Papers/modal-logic/

# Ingest from Zotero (requires zotero-library.json)
/literature --ingest --zotero "BlackburnDeRijkeVenema2001"

# Ingest and skip local loading prompt
/literature --ingest ~/Papers/modal-logic.pdf --no-local

# Ingest and automatically load into specs/literature/
/literature --ingest ~/Papers/modal-logic.pdf --local

Mode: Status (Default)

Show health report: processed vs unprocessed files and index.json state.

Status Step 1: Check Directory

if [ ! -d "$lit_dir" ]; then
  echo "## Literature Status"
  echo ""
  echo "No specs/literature/ directory found."
  echo "Create it and add PDF/DJVU files to get started."
  echo ""
  echo "**Tool Availability**:"
  echo "- pdftotext: $has_pdftotext"
  echo "- djvutxt: $has_djvutxt ($([ "$has_djvutxt" = "no" ] && echo 'install: nix-env -iA nixpkgs.djvulibre' || echo 'available'))"
  exit 0
fi

Status Step 2: Scan for Files

# Find all PDF and DJVU source files
pdf_files=$(find "$lit_dir" -name "*.pdf" 2>/dev/null | sort)
djvu_files=$(find "$lit_dir" -name "*.djvu" 2>/dev/null | sort)
all_source_files="$pdf_files $djvu_files"

# Find all markdown files (excluding any in subdirectory source_files/)
md_files=$(find "$lit_dir" -name "*.md" -not -path "*/source_files/*" 2>/dev/null | sort)

Status Step 3: Read Index

if [ -f "$index_file" ]; then
  entry_count=$(jq '.entries | length' "$index_file" 2>/dev/null || echo "0")
  indexed_paths=$(jq -r '.entries[].path' "$index_file" 2>/dev/null || echo "")
else
  entry_count=0
  indexed_paths=""
fi

Status Step 4: Compute Counts

# Count source files
pdf_count=$(echo "$pdf_files" | grep -c "\.pdf$" 2>/dev/null || echo 0)
djvu_count=$(echo "$djvu_files" | grep -c "\.djvu$" 2>/dev/null || echo 0)
md_count=$(echo "$md_files" | grep -c "\.md$" 2>/dev/null || echo 0)

# Identify unprocessed source files (PDFs/DJVUs without corresponding .md)
unprocessed=()
for src in $pdf_files $djvu_files; do
  basename_no_ext=$(basename "$src" | sed 's/\.[^.]*$//')
  # Check if any .md file starts with this basename
  if ! find "$lit_dir" -name "${basename_no_ext}*.md" -not -path "*/source_files/*" 2>/dev/null | grep -q .; then
    unprocessed+=("$src")
  fi
done
unprocessed_count=${#unprocessed[@]}
processed_count=$(( pdf_count + djvu_count - unprocessed_count ))

Status Step 5: Display Report

## Literature Status

**Directory**: specs/literature/
**Source Files**: {pdf_count} PDFs, {djvu_count} DJVUs
**Converted**: {processed_count} processed, {unprocessed_count} unprocessed
**Markdown Files**: {md_count}
**Index Entries**: {entry_count}

**Tool Availability**:
- pdftotext: {has_pdftotext}
- djvutxt: {has_djvutxt} {install hint if no}

{if unprocessed_count > 0}
**Unprocessed Files** ({unprocessed_count}):
- {file1}
- {file2}
...

Run `/literature --convert` to convert all, or `/literature --scan` to see details.
{end if}

{if entry_count > 0 and md_count != entry_count}
**Index Health**: {entry_count} indexed entries, {md_count} markdown files — run `/literature --validate` to check consistency.
{end if}

Mode: Scan

Find PDF/DJVU files lacking corresponding markdown conversions.

Scan Step 1: Check Directory

Same as Status Step 1 — exit gracefully if directory missing.

Scan Step 2: Find Unprocessed Files

unprocessed=()
for src in $(find "$lit_dir" -name "*.pdf" -o -name "*.djvu" 2>/dev/null | sort); do
  basename_no_ext=$(basename "$src" | sed 's/\.[^.]*$//')
  if ! find "$lit_dir" -name "${basename_no_ext}*.md" -not -path "*/source_files/*" 2>/dev/null | grep -q .; then
    unprocessed+=("$src")
  fi
done

Scan Step 3: Get Page Counts

For each unprocessed file, get page count via pdfinfo:

for src in "${unprocessed[@]}"; do
  ext="${src##*.}"
  if [ "$ext" = "pdf" ]; then
    if [ "$has_pdfinfo" = "yes" ]; then
      pages=$(pdfinfo "$src" 2>/dev/null | grep "^Pages:" | awk '{print $2}')
    else
      pages="unknown"
    fi
  elif [ "$ext" = "djvu" ]; then
    if [ "$has_djvutxt" = "yes" ]; then
      # djvused can get page count: djvused -e n file.djvu
      pages=$(djvused -e n "$src" 2>/dev/null || echo "unknown")
    else
      pages="unknown (djvutxt not installed)"
    fi
  fi
  echo "- $src ($pages pages)"
done

Scan Step 4: Display Results

## Literature Scan Results

**Unprocessed Files** ({count}):
- {file1} ({N} pages)
- {file2} ({N} pages)
...

**Tool Status**:
- pdftotext: {status}
- djvutxt: {status} {install hint if unavailable}

**Next Steps**:
- Convert all: `/literature --convert`
- Convert one: `/literature --convert path/to/file.pdf`

If no unprocessed files found:

## Literature Scan Results

All source files have been converted. No unprocessed PDFs or DJVUs found.

**Files**: {N} PDFs, {M} DJVUs — all converted
**Index**: {entry_count} entries in index.json

Run `/literature --validate` to check index.json consistency.

Mode: Validate

Check index.json against the filesystem for stale entries, missing files, and token count drift.

Validate Step 1: Load Index

if [ ! -f "$index_file" ]; then
  echo "## Literature Validation"
  echo ""
  echo "No index.json found at $index_file."
  echo "Run /literature to see status, or /literature --convert to convert files and create the index."
  exit 0
fi

# Iterate entries as WHOLE RECORDS (one compact-JSON object per line), never as bare
# `.path` strings. Driving the loop off `.entries[] | .path` collapses a `.path`-less
# entry (a schema-shape defect, e.g. a stub written by an older literature-ingest.sh)
# into the literal string "null" -- which then resolves to a nonexistent file
# "$lit_dir/null" and gets misreported as a missing FILE ("null (missing)") rather than
# surfaced as the schema defect it actually is. Reading each entry as a full record
# lets Step 2 below classify a path-less/id-less entry correctly before ever touching
# the filesystem.
entries=$(jq -c '.entries[]' "$index_file" 2>/dev/null)

Validate Step 2: Check Each Entry

For each indexed entry, check:

1. Existence at specs/literature/{entry.path}: -f for file-path entries, -d for directory-path entries (book/parent-level records whose path ends in /)
2. Token count drift (recount vs stored, flag if >20% different) — applies to file-path entries only; directory-path entries have no single content file to recount against
3. Required schema fields present: id, path, token_count, keywords, summary, doc_type, source_format
4. authors field shape: present and an array, all elements are strings, and no element looks like an unsplit comma-joined multi-author string (see authors-shape check below). This catches regressions from any future writer that reintroduces the malformed schema this check guards against — see .claude/context/project/literature/domain/literature-index.md for the tooling ownership boundary and .claude/scripts/literature-normalize-authors.sh for the companion fix-up tool.

stale_entries=()
drift_entries=()
schema_warnings=()
authors_shape_warnings=()
schema_shape_defects=()

while IFS= read -r entry_json; do
  [ -z "$entry_json" ] && continue

  entry_id=$(echo "$entry_json" | jq -r '.id // .doc_id // empty')
  entry_path=$(echo "$entry_json" | jq -r '.path // empty')

  # --- Schema-shape bucket, reported separately from stale entries ---
  # An entry missing .id (and .doc_id), missing .path, or carrying a .path that is not
  # sources/-prefixed is a SCHEMA-SHAPE defect (the entry itself is malformed), not a
  # missing-file defect (the entry is well-formed but its target vanished). Classifying
  # it here, before any filesystem check runs, is what ends the old "null (missing)"
  # misreport: a .path-less entry never reaches the file-existence check below at all.
  shape_defects=()
  [ -z "$entry_id" ] && shape_defects+=("missing .id/.doc_id")
  if [ -z "$entry_path" ]; then
    shape_defects+=("missing .path")
  elif [[ "$entry_path" != sources/* ]]; then
    shape_defects+=("path not sources/-prefixed: $entry_path")
  fi
  if [ "${#shape_defects[@]}" -gt 0 ]; then
    label="${entry_id:-<no id>}"
    joined=$(IFS=", "; echo "${shape_defects[*]}")
    schema_shape_defects+=("$label ($joined)")
    # A path-less entry has no filesystem target to check and no .path to key the
    # existing per-path checks below on -- skip straight to the next entry rather than
    # falling through into checks that assume entry_path is usable.
    [ -z "$entry_path" ] && continue
  fi

  full_path="$lit_dir/$entry_path"

  # Directory-path entries (trailing slash, or resolves to a directory on disk) are a
  # legitimate second schema variant: parent-level records for a book split into many
  # semantic chunks (doc_type: "book", token_count: 0 by design). They have no single
  # content file to recount tokens against, so existence is the sufficient check.
  if [[ "$entry_path" == */ ]] || [ -d "$full_path" ]; then
    if [ ! -d "$full_path" ]; then
      stale_entries+=("$entry_path (missing directory)")
      continue
    fi
  elif [ ! -f "$full_path" ]; then
    stale_entries+=("$entry_path (missing)")
    continue
  else
    # Recount tokens (file-path entries only)
    char_count=$(wc -c < "$full_path" 2>/dev/null || echo 0)
    current_tokens=$(( char_count / 4 + 20 ))
    stored_tokens=$(echo "$entry_json" | jq -r '.token_count // 0')

    if [ -n "$stored_tokens" ] && [ "$stored_tokens" -gt 0 ]; then
      # Calculate drift percentage
      diff=$(( current_tokens - stored_tokens ))
      if [ "$diff" -lt 0 ]; then diff=$(( -diff )); fi
      drift_pct=$(( diff * 100 / stored_tokens ))
      if [ "$drift_pct" -gt 20 ]; then
        drift_entries+=("$entry_path (stored: $stored_tokens, actual: $current_tokens, drift: ${drift_pct}%)")
      fi
    fi
  fi

  # Check for required schema fields (new in schema v2). Reads the already-parsed
  # entry record directly (no re-query by path needed), so this runs for every entry
  # that resolves on disk -- directory-path entries included.
  missing_fields=$(echo "$entry_json" | jq -r '
    [
      (if .doc_type == null or .doc_type == "" then "doc_type" else empty end),
      (if .source_format == null or .source_format == "" then "source_format" else empty end),
      (if .authors == null then "authors" else empty end),
      (if .title == null or .title == "" then "title" else empty end)
    ] | join(", ")
  ' 2>/dev/null || echo "")
  if [ -n "$missing_fields" ]; then
    schema_warnings+=("$entry_path (missing fields: $missing_fields)")
  fi

  # Check authors field shape (catch non-array or comma-joined authors so any
  # future writer that reintroduces the malformed schema is caught by routine validation).
  # The "possibly-comma-joined" heuristic is conservative: it flags an array element only
  # when it contains 2+ ", " occurrences, or exactly one ", " followed by a second
  # non-initial capitalized name-like token (2+ letters). This avoids false positives on
  # legitimate single-author "Last, First" or "Last, First M." formatting (the pattern the
  # former zotero index-add script itself produced), while still catching packed multi-author strings
  # like "Patrick Blackburn, Maarten de Rijke, Yde Venema" or two-author strings like
  # "Patrick Blackburn, Maarten de Rijke". Mirror this same heuristic in
  # literature-normalize-authors.sh so validate and normalize stay consistent. Also reads
  # the already-parsed entry record directly, so this covers directory-path entries too.
  authors_shape=$(echo "$entry_json" | jq -r '
    def is_comma_joined:
      ( [scan(", ")] | length ) as $n
      | if $n >= 2 then true
        elif $n == 1 then
          ( (split(", ")[1]) | ([scan("[A-Z][a-zA-Z]+")] | length) ) >= 2
        else false
        end;
    [
      (if .authors != null and (.authors | type) != "array" then "authors:not-array" else empty end),
      (if (.authors | type) == "array" and (.authors | any(type != "string")) then "authors:non-string-element" else empty end),
      (if (.authors | type) == "array" and (.authors | any(type == "string" and is_comma_joined)) then "authors:possibly-comma-joined" else empty end)
    ] | join(", ")
  ' 2>/dev/null || echo "")
  if [ -n "$authors_shape" ]; then
    authors_shape_warnings+=("$entry_path ($authors_shape)")
  fi
done <<< "$entries"

Validate Step 2b: Namespace Divergence Check (hard failure, with recorded exceptions)

Compares index.json's identity space against .literature.db's chunks_data.doc_id space, using the same path-derived bridge literature-search.sh's get_project_doc_ids() and literature-doc-key.sh already use (never .id alone — see that script's header for the full invariant). The corpus-side residue was reconciled (17 new parent entries added under their bare directory id, 2 duplicate ingest directories quarantined and their stub index entries removed — see context/project/literature/domain/literature-index.md's FTS-namespace subsection for the invariant and corpus history). This check now fails the command on any divergence entry outside the recorded known-exceptions list below; an empty divergence bucket (modulo those exceptions) is the expected steady state, not an aspiration.

Known exceptions (carried forward with a recorded reason, never silently expanded — adding to this list requires the same adjudication rigor as the original corpus reconciliation, not a quick edit to silence a new failure):

• gabbay_2000 (index-only: no FTS chunks) — conversion rejected; stamped provenance_fidelity: not_yet_converted. Not a defect to fix by this check; re-adjudicate by re-converting the source, not by editing this exceptions list.

# Validate mode can be invoked without passing through Convert mode's setup, so
# resolve SCRIPT_DIR defensively here rather than assuming it is already set.
SCRIPT_DIR="${SCRIPT_DIR:-$(dirname "$0")/../../scripts}"
DOC_KEY_SCRIPT="$SCRIPT_DIR/literature-doc-key.sh"
DIVERGENCE_KNOWN_EXCEPTIONS_NOTE="known exceptions: gabbay_2000 (index-only, conversion rejected, provenance_fidelity: not_yet_converted) -- any other divergence entry fails this check"
# Bare directory ids carried as recorded exceptions to the hard-failure gate below.
# Format matches the "{id} (dir key: {dir_key})" label divergence_index_only entries use.
DIVERGENCE_KNOWN_EXCEPTION_IDS=("gabbay_2000")

divergence_fts_only=()
divergence_index_only=()
divergence_id_inconsistent=()

if [ -x "$DOC_KEY_SCRIPT" ] && [ -f "$lit_dir/.literature.db" ] && command -v sqlite3 >/dev/null 2>&1; then
  fts_ids=$(sqlite3 "$lit_dir/.literature.db" "SELECT DISTINCT doc_id FROM chunks_data;" 2>/dev/null | sort -u)
  index_keys=$("$DOC_KEY_SCRIPT" --list-keys "$index_file" 2>/dev/null | sort -u)

  # FTS doc_ids with no index coverage at all (neither .id nor path-derived key reaches them)
  while IFS= read -r fid; do
    [ -z "$fid" ] && continue
    if ! grep -qxF "$fid" <<< "$index_keys"; then
      divergence_fts_only+=("$fid")
    fi
  done <<< "$fts_ids"

  # Index dir keys (parent entries' path-derived key) with no FTS chunks under that key
  while IFS= read -r entry_json; do
    [ -z "$entry_json" ] && continue
    is_parent=$(echo "$entry_json" | jq -r 'if (.parent_doc == null or .parent_doc == "") then "yes" else "no" end')
    [ "$is_parent" != "yes" ] && continue
    p_id=$(echo "$entry_json" | jq -r '.id // .doc_id // empty')
    p_path=$(echo "$entry_json" | jq -r '.path // empty')
    [ -z "$p_id" ] && continue
    dir_key="$p_id"
    if [[ "$p_path" == sources/* ]]; then
      dir_key="${p_path#sources/}"
      dir_key="${dir_key%%/*}"
    fi
    dir_key_in_fts="no"
    grep -qxF "$dir_key" <<< "$fts_ids" && dir_key_in_fts="yes"
    if [ "$dir_key_in_fts" = "no" ]; then
      divergence_index_only+=("$p_id (dir key: $dir_key)")
    fi
    # Parent entries whose .id is neither its own path-derived key nor present in FTS
    # under EITHER lookup strategy -- i.e. completely unreachable, not merely a
    # curated-id-differs-from-FTS-id pairing. The `.id != dir_key` paired case (the 17
    # supported entries Decision C names) is deliberately NOT reported here: when
    # `dir_key` itself resolves in FTS, the path bridge already covers that entry, and
    # Decision C is explicit that a differing curated `.id` is a supported
    # configuration in that case, not a defect. Gating on `dir_key_in_fts == no` (in
    # addition to `.id` also not resolving) is what keeps this bucket at the expected
    # near-zero count on a corpus where the bridge is doing its job, rather than
    # re-flagging every one of those 17 supported pairings as broken.
    if [ "$p_id" != "$dir_key" ] && [ "$dir_key_in_fts" = "no" ] && ! grep -qxF "$p_id" <<< "$fts_ids"; then
      divergence_id_inconsistent+=("$p_id (path-derived key: $dir_key)")
    fi
  done <<< "$entries"
  divergence_check_skipped="no"
else
  echo "Warning: namespace-divergence check skipped (literature-doc-key.sh, .literature.db, or sqlite3 unavailable)" >&2
  divergence_check_skipped="yes"
fi

# --- Split divergence_index_only into recorded-known-exceptions vs. unexpected ---
# (divergence_fts_only and divergence_id_inconsistent have no recorded exceptions today --
# every entry in either bucket is unexpected and fails the check. gabbay_2000 is the sole
# recorded exception, and it only ever lands in divergence_index_only.)
divergence_index_only_known=()
divergence_index_only_unexpected=()
for item in "${divergence_index_only[@]:-}"; do
  [ -z "$item" ] && continue
  is_known="no"
  for exc in "${DIVERGENCE_KNOWN_EXCEPTION_IDS[@]}"; do
    case "$item" in
      "$exc "*) is_known="yes"; break ;;
    esac
  done
  if [ "$is_known" = "yes" ]; then
    divergence_index_only_known+=("$item")
  else
    divergence_index_only_unexpected+=("$item")
  fi
done

# --- Hard-failure gate: any unexpected divergence (in any of the three buckets), or the
# check being skipped entirely (tools unavailable -- cleanliness cannot be confirmed,
# which is not the same as clean), fails Validate mode. See Step 4 for how this combines
# with the other failure classes (schema-shape defects, etc.) into the final report verdict.
divergence_check_failed="no"
if [ "$divergence_check_skipped" = "yes" ] \
   || [ "${#divergence_fts_only[@]}" -gt 0 ] \
   || [ "${#divergence_id_inconsistent[@]}" -gt 0 ] \
   || [ "${#divergence_index_only_unexpected[@]}" -gt 0 ]; then
  divergence_check_failed="yes"
fi

Validate Step 3: Find Unindexed Markdown Files

unindexed=()
while IFS= read -r md_file; do
  # Get path relative to lit_dir
  rel_path="${md_file#$lit_dir/}"
  if ! jq -e --arg p "$rel_path" '.entries[] | select(.path == $p)' "$index_file" >/dev/null 2>&1; then
    unindexed+=("$rel_path")
  fi
done < <(find "$lit_dir" -maxdepth 1 -name "*.md" 2>/dev/null | sort)

Validate Step 4: Display Report

## Literature Validation Report

**Index**: specs/literature/index.json
**Total Entries**: {N}

### Schema-Shape Defects ({count}) — malformed entries (missing .id/.doc_id, missing .path, or
### .path not sources/-prefixed) — reported separately from missing FILES below
{for each schema_shape_defect entry:}
- {label} ({defects})
  These entries are malformed records, not files that vanished; the fix is to repair or remove
  the entry, not to search the filesystem for a target.

### Stale Entries ({count}) — path in index but file missing
{for each stale entry:}
- {entry_path}

### Token Count Drift ({count}) — more than 20% change from stored count
{for each drift entry:}
- {entry_path}: stored {N}, actual {M} ({pct}% drift)

### Schema Warnings ({count}) — entries missing required v2 fields
{for each schema_warning entry:}
- {entry_path}: {missing_fields}
  Run: /literature --index {file_path} to update entry with missing fields

### Authors Shape Warnings ({count}) — non-array or comma-joined authors
{for each authors_shape_warning entry:}
- {entry_path}: {authors_shape}
  Run: bash .claude/scripts/literature-normalize-authors.sh {index_file} --apply to normalize,
  or run with no flag (dry-run is the default) first to preview the change.

### Namespace Divergence (hard failure — index.json vs. .literature.db chunks_data.doc_id)
{DIVERGENCE_KNOWN_EXCEPTIONS_NOTE}

#### FTS-only doc_ids with no index coverage ({count}) — always unexpected, always fails
{for each divergence_fts_only entry:}
- {doc_id}

#### Index dir keys with no FTS chunks — recorded known exceptions ({count}, does not fail)
{for each divergence_index_only_known entry:}
- {id} (dir key: {dir_key})

#### Index dir keys with no FTS chunks — unexpected ({count}, fails)
{for each divergence_index_only_unexpected entry:}
- {id} (dir key: {dir_key})

#### Parent entries whose .id is neither its own path-derived key nor present in FTS ({count}) — always unexpected, always fails
{for each divergence_id_inconsistent entry:}
- {id} (path-derived key: {dir_key})

{if divergence_check_skipped == "yes":}
**Namespace divergence check SKIPPED** (literature-doc-key.sh, .literature.db, or sqlite3
unavailable) — cleanliness cannot be confirmed. A skipped check counts as a failure below; it is
not treated as passing.

This section fails the command on any entry outside the recorded known-exceptions list above
(divergence_check_failed). The bridge (path-derived key resolution) already covers most
divergence at read time; a bucket entry that survives the bridge and is not a recorded exception
is exactly the residue this check exists to catch — it is not informational.

### Unindexed Files ({count}) — markdown files not in index.json
{for each unindexed file:}
- {file_path}
  Run: /literature --index {file_path}

{if all clean (zero schema-shape defects AND divergence_check_failed == "no" AND zero stale
entries AND zero unindexed files):}
### Validation Passed

All {N} index entries are valid. No schema-shape defects, no stale paths, no drift, no schema
warnings, no authors-shape warnings, no namespace divergence beyond the recorded known
exceptions, no unindexed files.

{else:}
### Validation FAILED

One or more checks above did not pass — see the sections with a non-zero unexpected count.
Namespace divergence beyond the recorded known exceptions is the specific defect class this
task exists to end; do not add an entry to the known-exceptions list to silence a new failure
without the same adjudication rigor the original corpus reconciliation used (provenance-based,
byte-identical-content verification, not a guess). Fix the underlying entry (add a missing
parent entry, quarantine a duplicate, or repair a schema-shape defect) and re-run.

Mode: Convert

Convert unprocessed PDF/DJVU files to markdown with interactive confirmation.

Convert Step 1: Determine Target Files

if [ -n "$file" ]; then
  # Convert specific file
  targets=("$file")
else
  # Find all unprocessed files
  targets=()
  for src in $(find "$lit_dir" -name "*.pdf" -o -name "*.djvu" 2>/dev/null | sort); do
    basename_no_ext=$(basename "$src" | sed 's/\.[^.]*$//')
    if ! find "$lit_dir" -name "${basename_no_ext}*.md" -not -path "*/source_files/*" 2>/dev/null | grep -q .; then
      targets+=("$src")
    fi
  done
fi

Convert Step 2: Check Tool Availability

if [ "$has_pdftotext" = "no" ]; then
  echo "Error: pdftotext not found. Install with: nix-env -iA nixpkgs.poppler_utils"
  exit 1
fi

Convert Step 3: Process Each File

For each target file:

3a: Get Page Count

src="$target_file"
ext="${src##*.}"
basename_no_ext=$(basename "$src" | sed 's/\.[^.]*$//')

if [ "$ext" = "djvu" ]; then
  if [ "$has_djvutxt" = "no" ]; then
    echo "Skipping $src: djvutxt not installed. Install with: nix-env -iA nixpkgs.djvulibre"
    continue
  fi
  # Get page count for DJVU
  page_count=$(djvused -e n "$src" 2>/dev/null || echo 1)
else
  # PDF: get page count
  if [ "$has_pdfinfo" = "yes" ]; then
    page_count=$(pdfinfo "$src" 2>/dev/null | grep "^Pages:" | awk '{print $2}')
  else
    page_count=1
  fi
fi

3b: Extract Full Text and Determine Chunking

First, extract the complete text from the source file (page-range extraction happens at 3d if needed for page-range chunks; for content-aware chunking, extract all text first):

if [ "$ext" = "pdf" ]; then
  full_text=$(pdftotext -layout "$src" - 2>/dev/null)
elif [ "$ext" = "djvu" ]; then
  full_text=$(djvutxt "$src" 2>/dev/null)
fi

# Count total lines
total_lines=$(echo "$full_text" | wc -l)
LINE_THRESHOLD=4000
MERGE_MIN=500

Content-aware chunking algorithm:

# Step 1: Detect logical section boundaries using heading patterns
# Supported heading patterns (in priority order):
#   - "Chapter N" / "CHAPTER N"  -> chapter boundary
#   - "N  Title" (number + spaces + capitalized text) -> numbered section
#   - "Part I/V/X..." / "Part 1/2..." -> part boundary
#   - "## Heading" / "### Heading" (markdown headings) -> section heading

section_starts=()  # line numbers where sections begin
section_names=()   # human-readable name for each section

while IFS= read -r line_num_and_content; do
  line_num="${line_num_and_content%%:*}"
  content="${line_num_and_content#*:}"
  if echo "$content" | grep -qE '^(Chapter|CHAPTER)[[:space:]]+[0-9IVXivx]+'; then
    section_starts+=("$line_num")
    section_names+=("$(echo "$content" | sed 's/^[[:space:]]*//' | cut -c1-60)")
  elif echo "$content" | grep -qE '^[0-9]+[[:space:]]{2,}[A-Z]'; then
    section_starts+=("$line_num")
    section_names+=("$(echo "$content" | sed 's/^[[:space:]]*//' | cut -c1-60)")
  elif echo "$content" | grep -qE '^Part[[:space:]]+([IVXivx]+|[0-9]+)'; then
    section_starts+=("$line_num")
    section_names+=("$(echo "$content" | sed 's/^[[:space:]]*//' | cut -c1-60)")
  elif echo "$content" | grep -qE '^#{1,3}[[:space:]]+\S'; then
    section_starts+=("$line_num")
    section_names+=("$(echo "$content" | sed 's/^#{1,3}[[:space:]]*//' | cut -c1-60)")
  fi
done < <(echo "$full_text" | grep -n "")

# Step 2: If headings found, merge small adjacent sections
if [ "${#section_starts[@]}" -gt 0 ]; then
  # Build merged chunks: combine adjacent sections until total lines >= LINE_THRESHOLD
  merged_chunks=()   # array of "start_line:end_line:name" strings
  chunk_start="${section_starts[0]}"
  chunk_name="${section_names[0]}"
  chunk_lines=0
  
  for i in "${!section_starts[@]}"; do
    if [ "$i" -eq 0 ]; then continue; fi
    prev_start="${section_starts[$((i-1))]}"
    curr_start="${section_starts[$i]}"
    section_size=$(( curr_start - prev_start ))
    
    if [ "$(( chunk_lines + section_size ))" -lt "$MERGE_MIN" ] || \
       [ "$(( chunk_lines + section_size ))" -lt "$LINE_THRESHOLD" ]; then
      # Merge into current chunk
      chunk_lines=$(( chunk_lines + section_size ))
    else
      # Flush current chunk
      chunk_end=$(( curr_start - 1 ))
      merged_chunks+=("${chunk_start}:${chunk_end}:${chunk_name}")
      chunk_start="$curr_start"
      chunk_name="${section_names[$i]}"
      chunk_lines=0
    fi
  done
  # Flush last chunk
  merged_chunks+=("${chunk_start}:${total_lines}:${chunk_name}")

  # Build chunks and output_files arrays from merged_chunks
  chunks=()
  output_files=()
  chunk_dir="$lit_dir/${sources_prefix}${basename_no_ext}"
  mkdir -p "$chunk_dir"
  
  for i in "${!merged_chunks[@]}"; do
    entry="${merged_chunks[$i]}"
    start_line="${entry%%:*}"
    rest="${entry#*:}"
    end_line="${rest%%:*}"
    name="${rest#*:}"
    slug=$(echo "$name" | tr '[:upper:]' '[:lower:]' | tr -cs 'a-z0-9' '-' | sed 's/^-//;s/-$//' | cut -c1-40)
    nn=$(printf "%02d" $(( i + 1 )))
    chunks+=("lines:${start_line}-${end_line}")
    output_files+=("${sources_prefix}${basename_no_ext}/section${nn}_${slug}.md")
  done

# Step 3: Fallback — no headings detected, use mechanical 4000-line splits
else
  chunks=()
  output_files=()
  start=1
  part_num=1
  chunk_dir="$lit_dir/${sources_prefix}${basename_no_ext}"
  
  if [ "$total_lines" -le "$LINE_THRESHOLD" ]; then
    # Single file — no chunking needed
    chunks+=("lines:1-${total_lines}")
    output_files+=("${sources_prefix}${basename_no_ext}.md")
  else
    mkdir -p "$chunk_dir"
    while [ "$start" -le "$total_lines" ]; do
      end=$(( start + LINE_THRESHOLD - 1 ))
      if [ "$end" -gt "$total_lines" ]; then end=$total_lines; fi
      nn=$(printf "%02d" "$part_num")
      chunks+=("lines:${start}-${end}")
      output_files+=("${sources_prefix}${basename_no_ext}/${basename_no_ext}_part${nn}.md")
      start=$(( end + 1 ))
      part_num=$(( part_num + 1 ))
    done
  fi
fi

3c: Confirm Chunk Boundaries with User

If multi-chunk (more than one output file), present proposed boundaries via AskUserQuestion:

Build a description showing detected sections or line ranges:

# Build display string from chunks and output_files arrays
chunk_preview=""
for i in "${!chunks[@]}"; do
  range="${chunks[$i]#lines:}"  # strip "lines:" prefix for display
  name=$(basename "${output_files[$i]}" .md)
  chunk_preview="${chunk_preview}\n  ${name}: lines ${range}"
done
approx_tokens=$(( total_lines * 15 / 10 ))  # rough estimate: 1.5 tokens/line

{
  "question": "Convert '{basename}' ({total_lines} lines) into {N} chunks?",
  "header": "Chunk Boundaries for {basename}",
  "multiSelect": false,
  "options": [
    {
      "label": "Accept proposed chunks ({N} files)",
      "description": "Detected sections:\n{chunk_preview}"
    },
    {
      "label": "Use single file (no chunking)",
      "description": "Convert all {total_lines} lines to one {basename}.md (~{approx_tokens} tokens)"
    },
    {
      "label": "Skip this file",
      "description": "Do not convert {basename} now"
    }
  ]
}

If user selects "Use single file": set chunks=("lines:1-${total_lines}"), output_files=("${basename_no_ext}.md") If user selects "Skip this file": continue to next file

3d: Write Chunk Files

For each chunk, extract the relevant lines from full_text and write to the output file:

for i in "${!chunks[@]}"; do
  chunk_range="${chunks[$i]#lines:}"  # strip "lines:" prefix
  start_line="${chunk_range%-*}"
  end_line="${chunk_range#*-}"
  output_md="$lit_dir/${output_files[$i]}"

  # Ensure parent directory exists (for chunked documents in subdirectory)
  mkdir -p "$(dirname "$output_md")"

  # Extract line range from full_text
  raw_text=$(echo "$full_text" | sed -n "${start_line},${end_line}p")

  # Check if text was extracted
  if [ -z "$(echo "$raw_text" | tr -d '[:space:]')" ]; then
    echo "Warning: No text in $src lines ${start_line}-${end_line}. File may be scanned/image-only and requires OCR."
    continue
  fi

  # Build title for this chunk
  doc_title=$(basename "$src" | sed 's/\.[^.]*$//' | tr '_-' '  ' | sed 's/\b\(.\)/\u\1/g')
  section_name=$(basename "$output_md" .md | sed 's/^[^_]*_//' | tr '-_' '  ')
  chunk_header=""
  if [ "${#chunks[@]}" -gt 1 ]; then
    chunk_header=" — ${section_name} (lines ${start_line}-${end_line})"
  fi

  markdown_content="# ${doc_title}${chunk_header}

${raw_text}"

  # Write to file
  echo "$markdown_content" > "$output_md"
done

3e: Compute Token Count and Auto-Generate Metadata

After writing each chunk file:

output_md="$lit_dir/${output_files[$i]}"
char_count=$(wc -c < "$output_md" 2>/dev/null || echo 0)
token_count=$(( char_count / 4 + 20 ))

# Extract auto-generated keywords (word frequency, top 10 after stopword removal)
# Stopword list (minimal)
stopwords="the a an and or but in on at to of for is are was were be been being have has had do does did will would could should may might shall can"

# Get word frequencies, filter stopwords, take top 10
auto_keywords=$(echo "$raw_text" | \
  tr '[:upper:]' '[:lower:]' | \
  tr -cs 'a-z' '\n' | \
  grep -v '^$' | \
  grep -v -w -F "$(echo "$stopwords" | tr ' ' '\n')" | \
  grep -E '^[a-z]{4,}$' | \
  sort | uniq -c | sort -rn | head -10 | \
  awk '{print $2}' | \
  jq -R . | jq -s . 2>/dev/null || echo '[]')

# Extract summary: look for Abstract, else use first 2-3 sentences
abstract_match=$(echo "$raw_text" | grep -i -A 5 "^[[:space:]]*abstract[[:space:]]*$" | head -6 | tail -5)
if [ -n "$abstract_match" ]; then
  auto_summary="$(echo "$abstract_match" | tr '\n' ' ' | sed 's/  */ /g' | cut -c1-300)"
else
  # First 2-3 sentences
  auto_summary=$(echo "$raw_text" | tr '\n' ' ' | sed 's/  */ /g' | grep -oP '^.{0,300}[.!?]' | head -1)
  if [ -z "$auto_summary" ]; then
    auto_summary=$(echo "$raw_text" | tr '\n' ' ' | sed 's/  */ /g' | cut -c1-200)
  fi
fi

3f: Confirm Metadata with User

First prompt for bibliographic fields:

{
  "question": "Enter bibliographic metadata for '{output_filename}' (or press Enter to skip each):",
  "header": "Document Metadata"
}

Prompt for each field in sequence using AskUserQuestion:

• {"question": "Authors (comma-separated, e.g. 'Alice Smith, Bob Jones'):"} -> parse into string array
• {"question": "Title (full document title):"} -> string
• {"question": "Year (publication year, e.g. 2024):"} -> integer or null
• {"question": "Document type (paper/book/chapter/section) [default: paper]:"} -> one of paper|book|chapter|section
• {"question": "Source format (pdf/djvu/manual) [auto-detected: {detected_format}]:"} -> one of pdf|djvu|manual (default to detected extension)

Then confirm keywords and summary:

{
  "question": "Review auto-generated keywords and summary for '{output_filename}':",
  "header": "Keywords and Summary",
  "multiSelect": false,
  "options": [
    {
      "label": "Accept auto-generated metadata",
      "description": "Keywords: {auto_keywords_preview}\nSummary: {auto_summary_preview}"
    },
    {
      "label": "Edit keywords",
      "description": "Keep summary, modify keyword list"
    },
    {
      "label": "Edit summary",
      "description": "Keep keywords, modify summary"
    },
    {
      "label": "Edit both",
      "description": "Modify both keywords and summary before indexing"
    }
  ]
}

If user selects "Edit keywords", prompt:

{"question": "Enter keywords (comma-separated):"}

Parse response into JSON array.

If user selects "Edit summary", prompt:

{"question": "Enter one-sentence summary:"}

3g: Update index.json

# Generate entry ID from filename (lowercase, underscores)
# For chunked sections, include subdirectory prefix to ensure uniqueness
if [[ "${output_files[$i]}" == *"/"* ]]; then
  entry_id=$(echo "${output_files[$i]}" | sed 's/\.md$//' | tr '[:upper:]' '[:lower:]' | tr '/ -' '_')
else
  entry_id=$(basename "$output_md" .md | tr '[:upper:]' '[:lower:]' | tr ' -' '_')
fi

# Determine source format from file extension
source_format="${ext}"  # "pdf" or "djvu"

# Determine doc_type, parent_doc, and page_range for chunked vs single-file entries
if [ "${#chunks[@]}" -gt 1 ] && [[ "${output_files[$i]}" == *"/"* ]]; then
  # Chunked section entry
  final_doc_type="section"
  parent_id=$(echo "$basename_no_ext" | tr '[:upper:]' '[:lower:]' | tr ' -' '_')
  parent_doc="$parent_id"
  chunk_range_display="${chunks[$i]#lines:}"
  page_range="lines:${chunk_range_display}"
else
  # Top-level (single file or user chose no-chunk) — use values from user prompt
  # final_doc_type already set from user prompt (default "paper")
  parent_doc=""
  page_range=""
fi

# Create or update index.json
if [ ! -f "$index_file" ]; then
  echo '{"token_budget": 4000, "entries": []}' > "$index_file"
fi

# Check if entry already exists
if jq -e --arg id "$entry_id" '.entries[] | selec

…(truncated)