theheavenlyd3mon Gutenberg — Public Domain Book Toolkit
Search, download, and extract clean text from Project Gutenberg — 70,000+ free public-domain ebooks. Ships a portable Python CLI with zero external dependencies.
Quick Start
# Search for books
python3 scripts/gutenberg search "Moby Dick"
# Download by Gutenberg ID (plain text)
python3 scripts/gutenberg download 2701 --format txt
# Download EPUB (for illustrated books)
python3 scripts/gutenberg download 2701 --format epub
# Extract clean text (strips PG boilerplate)
python3 scripts/gutenberg extract 2701
# Classify fiction vs non-fiction
python3 scripts/gutenberg classify 2701
# Full pipeline: search → download → extract
python3 scripts/gutenberg pipeline "Alice's Adventures in Wonderland"
How It Works
Project Gutenberg provides 70,000+ free public-domain ebooks in multiple formats. The gutendex API (https://gutendex.com) offers a free, unauthenticated JSON catalog. No API key required — just curl or this CLI.
Data Flow
User provides title/ID/author
↓
gutendex API search → pick book by ID
↓
Download plain text (preferred) or EPUB (fallback for illustrated books)
↓
Strip PG boilerplate → clean text
↓
Classify fiction/non-fiction → extract content
CLI Reference
search — Find books by keyword
python3 scripts/gutenberg search "Moby Dick"
python3 scripts/gutenberg search "Dracula" --limit 5
python3 scripts/gutenberg search "Sherlock Holmes" --json
python3 scripts/gutenberg search "Alice" --language en
Returns: ID, title, author (with life dates), language, subjects, download count. Results sorted by download count (most popular first).
metadata — Get full metadata for a book by ID
python3 scripts/gutenberg metadata 2701 # Moby Dick
python3 scripts/gutenberg metadata 11 # Alice's Adventures
python3 scripts/gutenberg metadata 1342 # Pride and Prejudice
python3 scripts/gutenberg metadata 1342 --json # JSON-only output
Returns: title, author(s), language(s), subjects, bookshelves, summaries, copyright status, download count, and all available format URLs.
download — Download a book by Gutenberg ID
# Plain text (UTF-8, preferred — works for most books)
python3 scripts/gutenberg download 2701 --format txt
# EPUB with images (for illustrated/scientific books)
python3 scripts/gutenberg download 2701 --format epub
# HTML (alternative fallback)
python3 scripts/gutenberg download 2701 --format html
# Specify output directory
python3 scripts/gutenberg download 2701 --format txt --output ./books/
The file is saved to ./gutenberg-<id>.<ext> (or --output path). Large books may take a moment.
extract — Strip PG boilerplate and produce clean text
python3 scripts/gutenberg extract 2701 # from downloaded txt
python3 scripts/gutenberg extract 2701 --input ./gutenberg-2701.txt
python3 scripts/gutenberg extract 2701 --format epub # extract from EPUB
Output: clean text without the Project Gutenberg license header/footer. For EPUB extraction (illustrated books), extracts text from all XHTML files and merges them into a single cleaned document.
Size detection: if a plain-text download is under 50KB for a known substantial book, warns that the text may be truncated and recommends EPUB mode.
classify — Classify fiction vs non-fiction
python3 scripts/gutenberg classify 2701
python3 scripts/gutenberg classify 2701 --json
Uses the book's subjects and bookshelves to classify:
- Fiction signals: "Fiction", "novels", "short stories", "poetry", "drama", "fantasy", "horror"
- Non-fiction signals: "Essays", "History", "Philosophy", "Biography", "Science", "Religion"
Returns: fiction, non-fiction, or ambiguous (with explanation of why).
pipeline — Full fetch pipeline
python3 scripts/gutenberg pipeline "Moby Dick" # search first
python3 scripts/gutenberg pipeline 2701 # by known ID
python3 scripts/gutenberg pipeline 2701 --clean /tmp/pipeline-output/ # save cleaned text
Runs: search (if title) → metadata → download (txt) → check size → extract (or EPUB fallback) → classify. Prints an executive summary at the end.
Global Flags
| Flag | Effect |
|---|---|
--json |
Output machine-readable JSON instead of human-readable text |
--quiet |
Suppress diagnostic output |
--dry-run |
Show what would be done without executing |
--output ./dir |
Save downloads to a specific directory |
--timeout 30 |
Override API timeout (default 15s) |
Fiction vs Non-Fiction Handling
When the classified result is fiction, the extracted text comes from an authored imagination. Consider splitting analysis into two tracks:
| Track | What it covers | Example claims |
|---|---|---|
| Canon | Facts within the fictional world — named entities, quoted lines, story events, world rules | "In Stoker's text, Dracula can assume wolf, bat, and mist forms" |
| Craft | Real-world technique — how the author achieves effect, publication history, literary influence | "Stoker's epistolary form forces the reader to piece together the narrative like an investigator" |
| Negative space | Deliberate omissions — what the author notably leaves unspecified | "Dracula is never granted interior voice in the novel" |
When classified as non-fiction, claims can be treated as real-world factual assertions about the subject matter.
Known Gotchas
- Plain text truncation for illustrated books — Books with diagrams, figures, or equations (geometry texts, scientific works, art books) may have plain-text downloads silently cut to 5-10KB (just the PG header). Always check file size. Under 50KB for a known substantial book → switch to EPUB extraction. The
pipelinecommand does this check automatically. - Gutendex can be slow or timeout — The API is a free service and can be slow for less popular books. The CLI uses a 15-second default timeout. Use
--timeout 30for slow responses, or navigate directly tohttps://www.gutenberg.org/ebooks/<id>as a fallback. - HTML downloads include navigation markup — HTML downloads contain site navigation and formatting. Prefer plain text or EPUB for clean text extraction.
- Rare books may 404 on certain format URLs — Not every book has every format. The CLI tries UTF-8 plain text first, falls back to US-ASCII, then to the
-0.txtfile path, then to EPUB, then to HTML. Thedownloadcommand reports which format was actually retrieved. - Rate limiting — Gutendex is unauthenticated but rate-limited. Batch requests with
sleep 1between calls for more than 10 rapid-fire requests. - utf-8 vs us-ascii — Gutendex returns both a
text/plain; charset=utf-8and atext/plain; charset=us-asciiURL. Prefer UTF-8; fall back to US-ASCII if the UTF-8 URL returns a 404. - Fiction classification ambiguity — Books with both fiction and non-fiction subjects (e.g. "Historical Fiction" + "History") are marked
ambiguous. Use--jsonto inspect the subject list and decide manually.
References
- scripts/gutenberg — Portable Python CLI. Zero external dependencies (stdlib only). Covers all major Gutenberg workflows: search, download (txt/epub/html), boilerplate stripping, EPUB text extraction, fiction classification, and the full pipeline.
- references/epub-extraction.md — EPUB text extraction details for illustrated books, with expanded Python walkthrough and format detection tips.
- Project Gutenberg — 70,000+ free ebooks.
- Gutendex API — JSON web API for the Project Gutenberg catalog.