Keyword Literature Download

Keyword Research Harvest

Use this skill when the user gives a topic or keyword set and wants a local literature-harvesting workflow, not a one-off manual search.

What this skill does

• Bundles its own literature_harvest/scripts stack, including search_pubmed.py, search_europepmc.py, search_crossref.py, search_openalex.py, merge_and_deduplicate.py, download_fulltexts.py, and harvest_utils.py.
• Searches PubMed/PMC, Europe PMC, Crossref, and OpenAlex through the bundled pipeline.
• Builds a no-dedup candidate table first.
• Downloads legal PDFs in parallel for faster collection.
• Lets the user specify the PDF output folder with --pdf-output-dir or download.pdf_output_dir.
• Keeps the PDF output folder PDF-only; HTML/XML helper files are cached under download_work/non_pdf_payloads/.
• Runs a second pass to chase PDF links from saved HTML pages.
• Produces a PDF-only deduplicated file folder and manifest after downloading.

Dependency model

This skill is self-contained. The target project does not need to already contain:

• literature_harvest/
• search_pubmed.py
• download_fulltexts.py

The bundled copies live under:

• literature_harvest/scripts/

Files in this skill

• scripts/run_keyword_harvest_no_dedup.py Use this to launch a new broad keyword harvest run into a new run folder.
• scripts/continue_download_and_dedup.py Use this to resume pending downloads, try HTML-to-PDF second pass, and build a deduplicated download set.
• literature_harvest/scripts/ Bundled search and download dependencies copied from a working local harvest stack so the skill is portable.
• references/config_template.json Copy and edit this for the topic-specific query set and filtering terms.
• references/prompt_template.md Reusable prompt for another AI/agent.

Recommended workflow

1. Choose any output parent folder where the new harvest run folder should be created.
2. Copy references/config_template.json and edit it for the user's topic keywords.
3. Run scripts/run_keyword_harvest_no_dedup.py with:
   • --output-root
   • --config
   • --run-name
   • optional --pdf-output-dir <folder> to put generated PDFs in a specific PDF-only folder
   • optional --download-workers <N> to tune parallel downloads
4. If the job is large, rerun with the same --run-name and --skip-search to avoid repeating API search.
5. Run scripts/continue_download_and_dedup.py --run-root <run-folder> --retry-failed; it reuses the stored PDF folder unless --pdf-output-dir is supplied again.
6. Report:
   • candidate count
   • PDF downloaded count
   • true PDF count
   • HTML/XML cache count
   • remaining pending
   • deduplicated keep count

Constraints

• Do not silently drop failed downloads.
• Keep metadata even when download fails.
• Prefer original research when the config asks for it, but do not hard-code one research domain.
• Do not claim HTML/XML is PDF.
• Never write HTML, XML, CSV, logs, manifests, or other helper files into the PDF output folder.
• Verify a file starts with the PDF signature before saving it into the PDF output folder.
• Treat second-pass cluster-like or support-like annotations as auxiliary only; downloading remains the primary task.

When to read references

• Read references/config_template.json before preparing a new run config.
• Read references/prompt_template.md when the user wants to hand this skill to another AI/agent.