Lisr PDF Downloader

LISR full-text collector

Collect full texts and supplements that the user is authorized to access, preserve a resumable record, and report what remains unavailable or needs user action.

Compose with installed skills

• Use browser:control-in-app-browser or the user's attached @Chrome session for LISR, LibKey, publisher, and institutional pages that require visible interaction or an authenticated session.
• Use spreadsheets:Spreadsheets to inspect the input workbook and verify the final status workbook.

Read references/browser-download-playbook.md before browser work.

Before running a command below, resolve <skill-dir> to the directory that contains this SKILL.md.

Non-negotiable boundaries

• Use visible page controls and offered download/export actions only.
• Never inspect or intercept bearer tokens, cookies, session storage, request headers, or hidden credentials.
• Never persist signed/bearer URL query strings. Send the polite-pool contact identity only to the named metadata APIs, never to a third-party PDF host.
• Never call undocumented authenticated endpoints or work around tool output/security filters.
• Never bypass a paywall, CAPTCHA, bot check, download restriction, license condition, or rate limit.
• Do not ask the user to give you institutional credentials. They log in themselves.
• Do not redistribute downloaded content or upload it to third-party services.
• Confirm that the intended use and access are permitted by the user's institution, publisher terms, and applicable license.

Inputs and naming

Obtain:

• ordered unique LISR study IDs and duplicates report;
• LISR project/workspace identity;
• institutional library/resolver identity if used;
• output and task-local state locations;
• a real contact email for APIs that require a polite-pool identity.

Store that user-confirmed address in CONTACT_EMAIL before running the API helpers. Do not copy a sample or institutional placeholder into this variable.

Metadata JSON must be an object keyed by the exact study ID. If a record also has an id field, it must equal its key. IDs may contain only safe filename characters, may not contain .., and are validated as a complete set before any study-derived path is constructed.

Name files:

• full text: <study-id>.pdf;
• one supplement: <study-id>_suppl.<ext>;
• multiple supplements: <study-id>_suppl1.<ext>, <study-id>_suppl2.<ext>, and so on.

Never overwrite an existing output. Investigate duplicates and retain the better verified copy deliberately.

Workflow

1. Prepare metadata through supported surfaces

Use LISR's visible search and any offered export function to obtain the record title, authors, identifier type/value, journal, date, trial registration, and abstract/full-text flags. If no export exists, capture only the needed visible fields. Match the exact study ID because search can be fuzzy.

Do not derive metadata by intercepting the site's authentication or calling its private backend.

2. Resolve identifiers

Run:

python3 <skill-dir>/scripts/resolve_identifiers.py \
  --meta lisr_metadata.json --out resolved.json --email "$CONTACT_EMAIL"

Review every resolved_doi_check candidate before downloading. A title match is not proof. Treat trial registrations and abstract-only records separately from journal full texts.

3. Open-access pilot

Run the open-access helper first, with a real, non-placeholder contact email:

python3 <skill-dir>/scripts/fetch_oa_and_supplements.py \
  --resolved resolved.json --out Downloaded_files \
  --results oa_results.json --email "$CONTACT_EMAIL"

The helper uses Unpaywall's OA result and Europe PMC supplements, bounds every response, permits only public HTTP(S) destinations and redirects, validates PDFs with an independent Poppler parse and validates archive structure, records the exact-DOI retrieval path, refuses output collisions, and pauses on rate limiting. A newly retrieved OA PDF remains identity_check until its internal title/DOI/authors are substantively matched to the study; an exact API record is not by itself content-identity proof. Open access permits retrieval, not unrestricted redistribution. The helper will not download a resolved_doi_check candidate until doi_verified is explicitly true with supporting evidence.

pdfinfo from Poppler must be available. If it is unavailable or rejects the file, the helpers fail closed and do not count the PDF as complete.

After reviewing a newly retrieved OA file, clear manual review only when the PDF itself matches the intended title, DOI, and authors. Set its OA state status to downloaded, retain oa_pdf: downloaded, set identity_verified: true, and record substantive identity_evidence. Otherwise leave status: identity_check; reruns intentionally do not clear it from stale evidence alone.

4. Pilot the institutional path before scaling

Choose one or two representative unresolved studies. In the user's signed-in browser:

1. open the LISR record or library resolver through the visible interface;
2. follow the offered full-text/LibKey link;
3. click the visible Download PDF or equivalent control;
4. wait for the browser's normal download to complete;
5. verify PDF structure and article identity against title/DOI/authors;
6. record the exact downloaded filename in a round manifest;
7. move it with scripts/move_round.py.

Confirm the pilot works, that the user is entitled to the content, and that the site/institution permits the proposed rate before continuing.

5. Continue in small visible-UI rounds

Use small batches and one completed browser download at a time unless the site explicitly supports batch export. Do not synthesize direct publisher PDF URLs, inject page JavaScript, or automate hidden download endpoints.

For each round, create a manifest naming only the files that were actually downloaded. Then run:

python3 <skill-dir>/scripts/move_round.py \
  --manifest round.json --downloads "/path/to/browser/downloads" \
  --out Downloaded_files --resolved resolved.json \
  --results browser_results.json --library 964 --batch 5

The mover never scans or cleans the Downloads folder. It handles only exact manifest filenames, validates full-text and supplement structure, requires a substantive identity/relationship attestation, and refuses destination collisions. For full text, include "identity_verified": true and "identity_evidence": "..." in the file entry. For supplements, use relationship_verified and relationship_evidence.

6. Handle barriers honestly

• If the visible resolver says no institutional access, record no_access.
• If a page presents a CAPTCHA, bot check, or human-verification interstitial, stop. Record user_action_required and let the user proceed manually if they choose. Do not take over afterward by extracting cookies or replaying requests.
• If the server returns or displays a rate limit, record rate_limited, stop the batch, and resume only after the stated window.
• If terms or license language prohibit automation/bulk retrieval, stop scaling and provide a link/status list for user-led retrieval.
• If the article identity is uncertain, record identity_check and do not file it as complete.

The only manifest states are no_access, user_action_required, rate_limited, license_restriction, identity_check, publisher_issue, and retry. The state file records each status class as terminal, retryable, or manual_review; downloaded, identity-attested full text is complete. Do not invent new status strings.

7. Retrieve supplements conservatively

Use Europe PMC supplements when available. Otherwise, use only visible publisher links labeled supplementary material, appendix, data supplement, or supporting information. Download sequentially through offered controls and record exact filenames plus a relationship attestation in the manifest. Only structurally validated, attested supplements appear in the verified workbook column. A missing supplement does not invalidate an otherwise verified main PDF; report it separately.

8. Build and inspect the status workbook

First call the workspace-dependency loader. Use only the Node executable and node_modules directory it returns; do not install or discover replacements. Then run:

python3 <skill-dir>/scripts/build_status.py \
  --out Downloaded_files --resolved resolved.json \
  --browser-results browser_results.json --oa-results oa_results.json \
  --xlsx Download_status.xlsx --library 964 \
  --node "$CODEX_NODE" --node-modules "$CODEX_NODE_MODULES" \
  --preview-dir status_previews

The wrapper creates a temporary writable builder directory and points its node_modules symlink only at the loader-provided dependency directory. It uses @oai/artifact-tool, inspects key ranges/formula errors, renders both sheets, and exports the workbook. Use view_image to inspect every retained preview. Confirm filters, unclipped content, plain-text URL cells, and formula-driven summary counts. Reconcile the workbook against actual output files and state JSON. Existing workbooks are not overwritten unless --overwrite is explicit.

Verification standard

Count a full text only when:

• the file passes supported header, cross-reference, and terminal EOF checks, and an independent Poppler parse confirms at least one readable page;
• title/DOI/authors match the intended study;
• state contains a substantive identity attestation and is not identity_check;
• the download came from an authorized OA or user-entitled institutional path;
• the expected output name is unambiguous;
• status state and output directory agree.

Keep source/access notes, license information when available, and manual-action reasons. Do not label an unresolved DOI or failed download as proof that no full text exists.

Scaling checklist

Before each larger round, confirm:

• the pilot remains valid on the relevant site;
• the batch size respects displayed limits and institutional/publisher terms;
• no rate-limit or verification warning appeared;
• previous files were verified and moved from an explicit manifest;
• destination collisions and duplicate studies were resolved;
• the user has not withdrawn or narrowed authorization.

When access conditions differ by publisher, treat each publisher as a new pilot rather than assuming the prior path applies.