# Research Source Acquisition

> Research Source Acquisition

- Skill: `lx050/research-source-acquisition` (Agent Skill, multi-file: 11 files)
- Install (CLI): `npx skillmds@latest add lx050/research-source-acquisition`
- Raw SKILL.md: https://api.skillmd.com/api/skills/lx050/research-source-acquisition/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- Author: Lx050 (https://skillmd.com/u/lx050)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/lx050/research-source-acquisition

---


# Research Source Acquisition

This Skill acquires online research sources in **two modes that are two stages of
one pipeline, never contradictions**. The direction is that the old strict path
now sits *behind* the funnel:

- **Mode one — funnel-intake (default, wide-in with accounting).** The research
  intake front door. Under owen's 2026-07-27 standing decision the collection
  side has no rights gate ("first fill a big funnel, then you can screen"). It
  fetches many sources into an immutable quarantine batch, logs every asset and
  every failure, and applies a default rights floor of `internal-research-only`.
  Engine: [`scripts/funnel_intake.py`](scripts/funnel_intake.py).
- **Mode two — publication-upgrade (strict, single approved asset).** The
  publish-side upgrade. After a human screening stage adjudicates a quarantined
  asset, mode two re-acquires exactly that one asset through the original gated
  path (D1+ rights record, exact-URL approval, SSRF-resistant checks, MIME/magic
  + SHA-256) so it is reproducibly frozen for a later approved surface. Engine:
  [`scripts/source_acquisition.py`](scripts/source_acquisition.py) (unchanged).

An optional discovery-only entry,
[`scripts/sciverse_candidate_adapter.py`](scripts/sciverse_candidate_adapter.py),
normalizes exported or live Sciverse SCP bibliographic results into untrusted
`candidate-envelope-v1` records. It is neither an acquisition mode nor a trusted
fact source; read
[`references/sciverse-candidate-adapter.md`](references/sciverse-candidate-adapter.md)
before using it.

## Two modes, one division of labor

| | **funnel-intake** (mode one, default) | **publication-upgrade** (mode two) |
|---|---|---|
| Role | research intake front door: wide-in, fetch first | publish-grade single-asset freeze after adjudication |
| Rights gate | **none at collection time** (funnel policy) | **required before download**: D1+ rights record, exact URL approved |
| Accounting | SHA-256 / source URL / UTC / observed-rights label / batch manifest, every asset and every failure | full provenance ledger + verification receipt + terms snapshot |
| Rights meaning | an **observed** metadata label only, never a fetch precondition | rights **precede** the fetch; three layers; `asset_url == --url` |
| Security | quarantine + per-asset accounting + polite pacing | SSRF/DNS-pinned IP / redirect allow-list / magic / framing |
| When | opening a new line of inquiry (multi-source intake) | promoting one screened asset toward a public/approved pipeline |

Both are real; neither replaces the other. Funnel-intake brings material in;
publication-upgrade freezes an adjudicated asset for release. What the funnel
exempts is only the *pre-collection* per-asset freeze — **not** the accounting,
and **not** the publication gate. The publication side is unchanged: the public
profile still ships **zero raster images** and every new asset defaults to
`internal-research-only`, out of the public bundle.

## Read the contract

Read [`references/funnel-intake-contract.md`](references/funnel-intake-contract.md)
before running mode one. Read
[`references/rights-record-contract.md`](references/rights-record-contract.md)
before creating a rights record or invoking the mode-two downloader; use
[`references/rights-record-v1.schema.json`](references/rights-record-v1.schema.json)
for authoring assistance and rely on the CLI for authoritative evidence checks.

---

## Mode one — funnel-intake (default)

Wide-in intake with an audit trail. Standard library only, offline-testable, and
it **never imports `source_acquisition.py`** — the two engines stay decoupled so
mode two's pinned bytes are untouched.

Give it absolute paths. `--adapters` is the adapter registry
(`api-acquisition-adapters-v01.json`, kept with the project's data registries);
`--batch-dir` is a gitignored quarantine directory; `--records-dir` is where the
manifest and receipt land.

```bash
# 1. plan: resolve fetchable adapters + a goal into a plan (blocked-* excluded)
python3 scripts/funnel_intake.py plan \
  --adapters /absolute/path/api-acquisition-adapters-v01.json \
  --goal /absolute/path/goal.json \
  --min-adapters 8 \
  --out-plan /absolute/path/plan.json

# 2. intake: fetch every planned target into a quarantine batch, log every outcome
python3 scripts/funnel_intake.py intake \
  --plan /absolute/path/plan.json \
  --batch-dir /absolute/path/quarantine/FUNNEL-INTAKE-XXX/ \
  --records-dir /absolute/path/records/FUNNEL-INTAKE-XXX/ \
  --batch-id FUNNEL-INTAKE-XXX \
  --resume

# 3. receipt: the machine-readable multi-source-floor verdict for the screening stage
python3 scripts/funnel_intake.py receipt \
  --records-dir /absolute/path/records/FUNNEL-INTAKE-XXX/ \
  --out /absolute/path/records/FUNNEL-INTAKE-XXX/intake-receipt.json

# manifest / verify recompute the batch manifest offline from bytes on disk
python3 scripts/funnel_intake.py verify \
  --records-dir /absolute/path/records/FUNNEL-INTAKE-XXX/ \
  --batch-dir /absolute/path/quarantine/FUNNEL-INTAKE-XXX/
```

Preserve these mode-one invariants:

- **Wide-in but accounted.** Every asset records SHA-256, source URL, UTC, the
  **observed** rights label, and a batch manifest. The rights label is metadata
  captured at intake, not a precondition for fetching it.
- **Default rights floor `internal-research-only`, `payload_status: quarantine`.**
  Intake is not clearance and not permission to publish.
- **The multi-source floor is honest.** `meets_multi_source_floor` is decided by
  `adapters_attempted >= min_adapters_required` **and**
  `independence_groups_spanned >= 2`. It is *attempted-and-logged*, **not**
  success-required: a batch where every source was blocked still meets the floor
  if enough independent adapters were attempted and their outcomes recorded.
  Failures (`429-exhausted`, `timeout`, `blocked`, `empty-manifest`) are
  first-class outcome rows, never omissions.
- **Default `--min-adapters` = every non-blocked adapter** in the registry (the
  `blocked-*` adapters are excluded from the plan but still counted), spanning at
  least two independence groups.
- **Polite pacing, 429 backoff, and redirect-follow are network etiquette, not
  gates.** Keep the declared User-Agent fixed, stop when an access-control
  challenge appears, and space retries conservatively.
- **Idempotent, resumable accounting.** A batch ID is never reused to overwrite
  bytes; `--resume` accepts already-present files instead of refetching. A
  committed batch README/OUTCOME/manifest is corrected by **append only**.
- **The screening stage is a separate human decision point.** `funnel_intake.py`
  puts material on the table; per-asset rights adjudication, upgrade, and any
  publication decision belong to that later stage. The receipt's
  `default_rights_floor` and `payload_status` are the handles it consumes.

---

## Mode two — publication-upgrade (strict)

Acquire exactly one approved source as untrusted bytes. Keep it in quarantine
until separate security, quality, and scholarly gates promote it.
Version 1.1.2 preserves verification compatibility with 1.1.0 and 1.1.1
acquisition manifests while emitting 1.1.2 for every new manifest and
verification receipt.

## Preserve these invariants

- Require a rights record with D1 or higher, `download: true`, an `asset_url`
  exactly equal to `--url`, and `approved_redirect_urls` as an explicit exact
  URL list.
- Bind every asset record to one absolute candidate-registry path, registry
  SHA-256, candidate ID, item page, structured downloadable URL, exact three
  rights layers, and a terms run for that same candidate.
- Permit only HTTPS and exact repeated `--allow-host` values. Never add
  wildcards or broaden the list merely because a redirect was blocked.
- Reject URL userinfo, credential-like query parameters, local hostnames, and
  every DNS result that is not globally routable.
- Default to zero redirects. When a reviewed redirect is necessary, require its
  exact URL in the rights record, re-resolve it, and pin the connection to a
  validated IP while retaining the hostname for TLS certificate validation.
- Disable environment proxies by using the direct HTTPS transport.
- Require determinate body framing on the final `200`. A body delimited only by
  connection close cannot be distinguished from a body the peer cut short, so a
  response with neither `Content-Length` nor a chunked transfer coding is
  blocked (`indeterminate-body-framing`, exit `3`) rather than published as
  complete. Declaring both is `ambiguous-body-framing` (exit `3`); any transfer
  coding other than `identity` or `chunked` is `unsupported-transfer-coding`
  (exit `5`). This is an acquisition-time gate only: manifests written before it
  existed still verify byte-identically.
- Stream into a newly created quarantine run, enforce header and body limits,
  hash while writing, and call `fsync`.
- Never execute, import, decompress, render, OCR, or otherwise interpret the
  payload as instructions. Metadata JSON is parsed only after a fixed-budget,
  no-AST syntax scan and bounded numeric conversion. Downloaded JSON format
  recognition uses a separate fixed-budget syntax scanner and never calls
  `json.loads` or constructs a complete payload value tree. Both paths refuse
  duplicate keys, non-finite values, BOMs, invalid UTF-8, excessive nesting,
  oversized numbers, and their respective resource limits.
- Compare rights-declared MIME, HTTP `Content-Type`, and detected magic/actual
  format. Keep mismatches as read-only `.blocked` evidence.
- Never reuse a run ID or overwrite a rights snapshot, payload, manifest, or
  sidecar.
- Require a canonical non-symlink quarantine root owned by the effective user
  and not writable by group or others. Create the run as exact mode `0700`
  under the same owner, then pin it and every ancestor by directory descriptor
  for the full transaction. Perform all run-local creation, append, read, stat,
  link, unlink, and publication descriptor-relatively. Refuse ancestry or
  run-entry identity drift, require single-link regular files and no-replace
  publication, and retain a two-event append-only run journal.
- Treat `status: quarantine` as acquisition success, not security clearance or
  permission to publish.

## Bootstrap rights evidence

When no local terms snapshot exists, use the contract's non-circular
`evidence-bootstrap` flow:

1. Pin the reviewed candidate registry by absolute path and actual local
   SHA-256.
2. Approve one HTML or JSON URL occurring in a structured rights field for the
   selected candidate.
3. Run `preflight`, `acquire`, and `verify` for the evidence URL.
4. Create the final `asset` rights record from that verified run and payload
   hash, repeating the same registry path/hash and candidate ID.

Do not hand-enter an unverifiable terms digest. The final asset record must
point to a successful local evidence run; the CLI re-verifies it before any
asset request.

## Preflight

This section and the ones below it belong to **mode two**
(publication-upgrade). Run preflight before every **mode-two** acquisition
(the funnel of mode one has no preflight gate — see "Mode one" above):

```bash
python3 scripts/source_acquisition.py preflight \
  --url "https://approved.example/path/asset.jpg" \
  --rights-record /absolute/path/rights-record.json \
  --allow-host approved.example \
  --timeout-seconds 15 \
  --max-redirects 0 \
  --max-header-bytes 65536 \
  --max-body-bytes 26214400
```

Preflight parses the rights record, verifies its evidence chain, checks the
exact URL and allow-list, resolves DNS, and validates limits. It does not fetch
the response body.

## Acquire

Use the same inputs and limits that passed preflight:

```bash
python3 scripts/source_acquisition.py acquire \
  --url "https://approved.example/path/asset.jpg" \
  --rights-record /absolute/path/rights-record.json \
  --allow-host approved.example \
  --quarantine-root /absolute/path/quarantine \
  --run-id acq-approved-asset-001 \
  --timeout-seconds 15 \
  --max-redirects 0 \
  --max-header-bytes 65536 \
  --max-body-bytes 26214400
```

Repeat `--allow-host` only for each independently approved redirect host, and
also list every exact redirect target in `approved_redirect_urls`. The command
emits its manifest and returns the manifest exit code. Run it through
`research-run-ledger`; preserve a nonzero run directory and `events.jsonl` as
audit evidence, and never feed a `.blocked` payload downstream.

## Verify

Recompute the complete offline contract: manifest sidecar and exact fields,
event journal, request and redirect chain, candidate and terms evidence,
rights snapshot, provider digest, payload hash/size/magic type, fixed filenames,
single-link files, and exact run-directory contents:

```bash
python3 scripts/source_acquisition.py verify \
  --run-directory /absolute/path/quarantine/acq-approved-asset-001
```

Run verification before registering the file in the provenance ledger and
again before any later promotion. Stop on exit code 6. A recomputed SHA sidecar
without the required acquisition and evidence semantics does not pass.
Verification of an `application/json` payload reuses the same bounded,
no-AST format scanner as acquisition; it is format recognition, not semantic
validation or downstream content admission.

When the run ledger or a later promotion gate needs a stable machine-readable
handoff, add an absolute path to a new receipt file in an already existing,
canonical directory:

```bash
python3 scripts/source_acquisition.py verify \
  --run-directory /absolute/path/quarantine/acq-approved-asset-001 \
  --receipt /absolute/path/run-evidence/source-verification.json
```

For a separately preregistered cross-process handoff, also pass the exact
unsigned decimal identities captured by the parent process:

```bash
python3 scripts/source_acquisition.py verify \
  --run-directory /absolute/path/quarantine/acq-approved-asset-001 \
  --run-directory-identity 16777234:123456 \
  --receipt /absolute/path/run-evidence/source-verification.json \
  --receipt-parent-identity 16777234:654321
```

Both identity guards use strict `DEV:INO` form. A mismatch is rejected before
offline verification reads the run payload or creates the receipt.
The verifier consumes the manifest, sidecar, event journal, rights snapshot,
payload, and directory listing through the pinned run descriptor for the full
offline verification. A transient pathname replacement cannot redirect those
reads to replacement bytes; persistent ancestry or run-entry drift blocks
before receipt publication.

The receipt must be outside the acquisition run and every descendant of that
run. The verifier pins the run and receipt-parent directory identities, opens
each parent component descriptor-relatively without following links, creates
the receipt exclusively, applies exact mode `0400`, and syncs both the file and
its parent directory. It refuses relative, non-canonical, symlink-traversing,
identity-changing, in-run, or existing output paths without overwriting them,
and emits the exact same JSON bytes to stdout for `research-run-ledger`
capture. It also preserves a receipt for an integrity mismatch, so receipt
existence is not admission: the ledger and every downstream gate must still
require process exit `0`, `ok: true`, and `status: verified`. Never retry by
deleting or overwriting an existing receipt; allocate a new append-only run or
correction artifact.

## Handoff

For an admitted payload:

1. Register the exact payload and manifest in the provenance ledger.
2. Run format-specific file-safety, resource-expansion, pixel/page/frame, and
   metadata-limit checks in an isolated process.
3. Keep original, deterministic derivative, and hypothetical reconstruction
   identities separate.
4. Promote only through an explicit later gate. Never copy quarantine directly
   into a public, model, OCR, or application directory.

See the contract for the AIC single-sample sequence, output fields, and exit
codes.