Research Source Acquisition

Research Source Acquisition

This Skill acquires online research sources in two modes that are two stages of one pipeline, never contradictions. The direction is that the old strict path now sits behind the funnel:

• Mode one — funnel-intake (default, wide-in with accounting). The research intake front door. Under owen's 2026-07-27 standing decision the collection side has no rights gate ("first fill a big funnel, then you can screen"). It fetches many sources into an immutable quarantine batch, logs every asset and every failure, and applies a default rights floor of internal-research-only. Engine: scripts/funnel_intake.py.
• Mode two — publication-upgrade (strict, single approved asset). The publish-side upgrade. After a human screening stage adjudicates a quarantined asset, mode two re-acquires exactly that one asset through the original gated path (D1+ rights record, exact-URL approval, SSRF-resistant checks, MIME/magic
  • SHA-256) so it is reproducibly frozen for a later approved surface. Engine: scripts/source_acquisition.py (unchanged).

An optional discovery-only entry, scripts/sciverse_candidate_adapter.py, normalizes exported or live Sciverse SCP bibliographic results into untrusted candidate-envelope-v1 records. It is neither an acquisition mode nor a trusted fact source; read references/sciverse-candidate-adapter.md before using it.

Two modes, one division of labor

Both are real; neither replaces the other. Funnel-intake brings material in; publication-upgrade freezes an adjudicated asset for release. What the funnel exempts is only the pre-collection per-asset freeze — not the accounting, and not the publication gate. The publication side is unchanged: the public profile still ships zero raster images and every new asset defaults to internal-research-only, out of the public bundle.

Read the contract

Read references/funnel-intake-contract.md before running mode one. Read references/rights-record-contract.md before creating a rights record or invoking the mode-two downloader; use references/rights-record-v1.schema.json for authoring assistance and rely on the CLI for authoritative evidence checks.

Mode one — funnel-intake (default)

Wide-in intake with an audit trail. Standard library only, offline-testable, and it never imports source_acquisition.py — the two engines stay decoupled so mode two's pinned bytes are untouched.

Give it absolute paths. --adapters is the adapter registry (api-acquisition-adapters-v01.json, kept with the project's data registries); --batch-dir is a gitignored quarantine directory; --records-dir is where the manifest and receipt land.

# 1. plan: resolve fetchable adapters + a goal into a plan (blocked-* excluded)
python3 scripts/funnel_intake.py plan \
  --adapters /absolute/path/api-acquisition-adapters-v01.json \
  --goal /absolute/path/goal.json \
  --min-adapters 8 \
  --out-plan /absolute/path/plan.json

# 2. intake: fetch every planned target into a quarantine batch, log every outcome
python3 scripts/funnel_intake.py intake \
  --plan /absolute/path/plan.json \
  --batch-dir /absolute/path/quarantine/FUNNEL-INTAKE-XXX/ \
  --records-dir /absolute/path/records/FUNNEL-INTAKE-XXX/ \
  --batch-id FUNNEL-INTAKE-XXX \
  --resume

# 3. receipt: the machine-readable multi-source-floor verdict for the screening stage
python3 scripts/funnel_intake.py receipt \
  --records-dir /absolute/path/records/FUNNEL-INTAKE-XXX/ \
  --out /absolute/path/records/FUNNEL-INTAKE-XXX/intake-receipt.json

# manifest / verify recompute the batch manifest offline from bytes on disk
python3 scripts/funnel_intake.py verify \
  --records-dir /absolute/path/records/FUNNEL-INTAKE-XXX/ \
  --batch-dir /absolute/path/quarantine/FUNNEL-INTAKE-XXX/

Preserve these mode-one invariants:

• Wide-in but accounted. Every asset records SHA-256, source URL, UTC, the observed rights label, and a batch manifest. The rights label is metadata captured at intake, not a precondition for fetching it.
• Default rights floor internal-research-only, payload_status: quarantine. Intake is not clearance and not permission to publish.
• The multi-source floor is honest. meets_multi_source_floor is decided by adapters_attempted >= min_adapters_required and independence_groups_spanned >= 2. It is attempted-and-logged, not success-required: a batch where every source was blocked still meets the floor if enough independent adapters were attempted and their outcomes recorded. Failures (429-exhausted, timeout, blocked, empty-manifest) are first-class outcome rows, never omissions.
• Default --min-adapters = every non-blocked adapter in the registry (the blocked-* adapters are excluded from the plan but still counted), spanning at least two independence groups.
• Polite pacing, 429 backoff, and redirect-follow are network etiquette, not gates. Keep the declared User-Agent fixed, stop when an access-control challenge appears, and space retries conservatively.
• Idempotent, resumable accounting. A batch ID is never reused to overwrite bytes; --resume accepts already-present files instead of refetching. A committed batch README/OUTCOME/manifest is corrected by append only.
• The screening stage is a separate human decision point. funnel_intake.py puts material on the table; per-asset rights adjudication, upgrade, and any publication decision belong to that later stage. The receipt's default_rights_floor and payload_status are the handles it consumes.

Mode two — publication-upgrade (strict)

Acquire exactly one approved source as untrusted bytes. Keep it in quarantine until separate security, quality, and scholarly gates promote it. Version 1.1.2 preserves verification compatibility with 1.1.0 and 1.1.1 acquisition manifests while emitting 1.1.2 for every new manifest and verification receipt.

Preserve these invariants

• Require a rights record with D1 or higher, download: true, an asset_url exactly equal to --url, and approved_redirect_urls as an explicit exact URL list.
• Bind every asset record to one absolute candidate-registry path, registry SHA-256, candidate ID, item page, structured downloadable URL, exact three rights layers, and a terms run for that same candidate.
• Permit only HTTPS and exact repeated --allow-host values. Never add wildcards or broaden the list merely because a redirect was blocked.
• Reject URL userinfo, credential-like query parameters, local hostnames, and every DNS result that is not globally routable.
• Default to zero redirects. When a reviewed redirect is necessary, require its exact URL in the rights record, re-resolve it, and pin the connection to a validated IP while retaining the hostname for TLS certificate validation.
• Disable environment proxies by using the direct HTTPS transport.
• Require determinate body framing on the final 200. A body delimited only by connection close cannot be distinguished from a body the peer cut short, so a response with neither Content-Length nor a chunked transfer coding is blocked (indeterminate-body-framing, exit 3) rather than published as complete. Declaring both is ambiguous-body-framing (exit 3); any transfer coding other than identity or chunked is unsupported-transfer-coding (exit 5). This is an acquisition-time gate only: manifests written before it existed still verify byte-identically.
• Stream into a newly created quarantine run, enforce header and body limits, hash while writing, and call fsync.
• Never execute, import, decompress, render, OCR, or otherwise interpret the payload as instructions. Metadata JSON is parsed only after a fixed-budget, no-AST syntax scan and bounded numeric conversion. Downloaded JSON format recognition uses a separate fixed-budget syntax scanner and never calls json.loads or constructs a complete payload value tree. Both paths refuse duplicate keys, non-finite values, BOMs, invalid UTF-8, excessive nesting, oversized numbers, and their respective resource limits.
• Compare rights-declared MIME, HTTP Content-Type, and detected magic/actual format. Keep mismatches as read-only .blocked evidence.
• Never reuse a run ID or overwrite a rights snapshot, payload, manifest, or sidecar.
• Require a canonical non-symlink quarantine root owned by the effective user and not writable by group or others. Create the run as exact mode 0700 under the same owner, then pin it and every ancestor by directory descriptor for the full transaction. Perform all run-local creation, append, read, stat, link, unlink, and publication descriptor-relatively. Refuse ancestry or run-entry identity drift, require single-link regular files and no-replace publication, and retain a two-event append-only run journal.
• Treat status: quarantine as acquisition success, not security clearance or permission to publish.

Bootstrap rights evidence

When no local terms snapshot exists, use the contract's non-circular evidence-bootstrap flow:

1. Pin the reviewed candidate registry by absolute path and actual local SHA-256.
2. Approve one HTML or JSON URL occurring in a structured rights field for the selected candidate.
3. Run preflight, acquire, and verify for the evidence URL.
4. Create the final asset rights record from that verified run and payload hash, repeating the same registry path/hash and candidate ID.

Do not hand-enter an unverifiable terms digest. The final asset record must point to a successful local evidence run; the CLI re-verifies it before any asset request.

Preflight

This section and the ones below it belong to mode two (publication-upgrade). Run preflight before every mode-two acquisition (the funnel of mode one has no preflight gate — see "Mode one" above):

python3 scripts/source_acquisition.py preflight \
  --url "https://approved.example/path/asset.jpg" \
  --rights-record /absolute/path/rights-record.json \
  --allow-host approved.example \
  --timeout-seconds 15 \
  --max-redirects 0 \
  --max-header-bytes 65536 \
  --max-body-bytes 26214400

Preflight parses the rights record, verifies its evidence chain, checks the exact URL and allow-list, resolves DNS, and validates limits. It does not fetch the response body.

Acquire

Use the same inputs and limits that passed preflight:

python3 scripts/source_acquisition.py acquire \
  --url "https://approved.example/path/asset.jpg" \
  --rights-record /absolute/path/rights-record.json \
  --allow-host approved.example \
  --quarantine-root /absolute/path/quarantine \
  --run-id acq-approved-asset-001 \
  --timeout-seconds 15 \
  --max-redirects 0 \
  --max-header-bytes 65536 \
  --max-body-bytes 26214400

Repeat --allow-host only for each independently approved redirect host, and also list every exact redirect target in approved_redirect_urls. The command emits its manifest and returns the manifest exit code. Run it through research-run-ledger; preserve a nonzero run directory and events.jsonl as audit evidence, and never feed a .blocked payload downstream.

Verify

Recompute the complete offline contract: manifest sidecar and exact fields, event journal, request and redirect chain, candidate and terms evidence, rights snapshot, provider digest, payload hash/size/magic type, fixed filenames, single-link files, and exact run-directory contents:

python3 scripts/source_acquisition.py verify \
  --run-directory /absolute/path/quarantine/acq-approved-asset-001

Run verification before registering the file in the provenance ledger and again before any later promotion. Stop on exit code 6. A recomputed SHA sidecar without the required acquisition and evidence semantics does not pass. Verification of an application/json payload reuses the same bounded, no-AST format scanner as acquisition; it is format recognition, not semantic validation or downstream content admission.

When the run ledger or a later promotion gate needs a stable machine-readable handoff, add an absolute path to a new receipt file in an already existing, canonical directory:

python3 scripts/source_acquisition.py verify \
  --run-directory /absolute/path/quarantine/acq-approved-asset-001 \
  --receipt /absolute/path/run-evidence/source-verification.json

For a separately preregistered cross-process handoff, also pass the exact unsigned decimal identities captured by the parent process:

python3 scripts/source_acquisition.py verify \
  --run-directory /absolute/path/quarantine/acq-approved-asset-001 \
  --run-directory-identity 16777234:123456 \
  --receipt /absolute/path/run-evidence/source-verification.json \
  --receipt-parent-identity 16777234:654321

Both identity guards use strict DEV:INO form. A mismatch is rejected before offline verification reads the run payload or creates the receipt. The verifier consumes the manifest, sidecar, event journal, rights snapshot, payload, and directory listing through the pinned run descriptor for the full offline verification. A transient pathname replacement cannot redirect those reads to replacement bytes; persistent ancestry or run-entry drift blocks before receipt publication.

The receipt must be outside the acquisition run and every descendant of that run. The verifier pins the run and receipt-parent directory identities, opens each parent component descriptor-relatively without following links, creates the receipt exclusively, applies exact mode 0400, and syncs both the file and its parent directory. It refuses relative, non-canonical, symlink-traversing, identity-changing, in-run, or existing output paths without overwriting them, and emits the exact same JSON bytes to stdout for research-run-ledger capture. It also preserves a receipt for an integrity mismatch, so receipt existence is not admission: the ledger and every downstream gate must still require process exit 0, ok: true, and status: verified. Never retry by deleting or overwriting an existing receipt; allocate a new append-only run or correction artifact.

Handoff

For an admitted payload:

1. Register the exact payload and manifest in the provenance ledger.
2. Run format-specific file-safety, resource-expansion, pixel/page/frame, and metadata-limit checks in an isolated process.
3. Keep original, deterministic derivative, and hypothetical reconstruction identities separate.
4. Promote only through an explicit later gate. Never copy quarantine directly into a public, model, OCR, or application directory.

See the contract for the AIC single-sample sequence, output fields, and exit codes.