Bundle maintenance
Open this when changing the manual bundle or checking whether a fact belongs in
a script, asset, or reference. The parent manual owns first-call behavior;
operation-contract.md owns exact Web semantics; the
tier index owns recovery routing.
Retained procedure support
| Path |
Owner |
Use |
scripts/extract_page.py |
executable helper |
Explicit --tier, --search, --fallback, and auto_tier() procedures. |
scripts/cached_get.py |
executable helper |
Legacy cache helper; currently broken on its normal successful-GET path (below). |
assets/api-endpoints.json |
endpoint data |
Academic, search, extraction, and real-time endpoint metadata. |
assets/css-selectors.json |
selector data |
Reusable extraction selectors. |
assets/site-templates.json |
site data |
Known-site selectors and endpoint hints. |
assets/extraction-pipeline.json |
pipeline data |
Legacy text: not valid JSON; not a runnable pipeline or Web policy. |
assets/regex-patterns.json |
identifier data |
DOI/arXiv/PMID/PMC/ISBN and URL patterns. |
assets/search-providers.json |
external provider data |
Explicit vendor recipes; verify availability and authorization. |
Scripts describe their own implementation only, not the public Web contract.
Assets are historical hints, not current authority: fixed free/unlimited/quota
claims, example contacts, package names and endpoints require upstream checking. Inspect current vendor documentation for endpoints, quotas,
dependencies, and prices; this bundle grants no install, credential, paid-use,
network, or access-control authority. Never put secrets or private paths in
examples or diagnostics.
Known legacy limitations
cached_get.py has no imported JSON module and its json argument shadows
that name. A normal successful GET reaches json.dump and raises
AttributeError after opening its cache file for writing; an existing file
may already have been truncated. Cache reads also fail. Do not recommend it
as a working cache or run it over shared/pre-existing data. Its cache key
includes URL/method but excludes params, headers and identity. No automatic
purge occurs: old claims about one-day/200-entry eviction were not implemented.
- Cache defaults are
/tmp/web-browsing-cache, overridden by
WEB_BROWSING_CACHE_DIR, and TTL 3600, overridden by
WEB_BROWSING_CACHE_TTL at import. clear_cache(url) unlinks that GET entry;
clear_cache() unlinks every matching JSON file. Both require explicit
deletion authorization; neither is a diagnostic. The CLI prints a preview.
assets/extraction-pipeline.json is not valid JSON (500/month is an
unquoted expression). Read it only as legacy notes, not machine configuration.
The other parseable assets are not automatically consumed by public Web.
extract_page.py is a separate Requests/browser/vendor helper, not the vetted
public BrowserEngine. Its preview fields are truncated (including the JSON
saved by --json); exit 0 means no error key, not complete text or useful
content. CrossRef helper is_oa actually contains a citation count, not an OA
boolean; verify OA against the real source. --save/--json overwrite paths.
--fallback can advance through local tiers to Jina, sending the target
URL to an external service. auto_tier is a heuristic; explicit --tier
still may do several metadata requests. Authorize the whole selected route,
dependencies and data disclosure before execution; no implicit chain.
Helper diagnostics may include raw URLs/exceptions: never feed secrets or
private URLs. Tier 3 blocks images/styles/fonts/media and returns only a
preview, so it is not a visual-validation harness.
These are preserved implementation limitations, not runtime fixes in the manual
reduction. For a proposed repair, use the lingtai-issue-report protocol and
obtain filing/implementation authority; do not silently patch installed helpers.
Cleanup / Footprint
Public Web leaves complete tmp/tool-results/web-* artifacts; snapshots,
cursors and references are bounded process-local state. Optional helpers leave
explicit downloads/JSON outputs and the cache described above. None are
implicitly disposable: preserve live referenced results, evidence, user data,
configuration and secrets, and never blindly clear a shared cache.
After a large retrieval session or before retirement, follow skills-manual →
reference/cleanup-footprint-contract.md#shared-footprint-check-recipe. Select
only this Agent's tmp/tool-results/web-* plus the exact task-created output
paths; inspect a configured helper cache separately only when authorized.
The shared recipe reports count/bytes without changing inspected files; its
optional logs/cleanup.jsonl append is a separate explicit write. Present a
dry-run listing what stays/goes, obtain explicit human consent, then perform
only the approved cleanup and record timestamp/tool/dry-run-or-apply/count/bytes/
path summary/approval. If consent is absent, stop at inspection.
1---2name: web-manual-maintenance-bundles3description: Inventory and ownership map for the bundled Web external procedures, assets, and retained helper scripts.4---5# Bundle maintenance67Open this when changing the manual bundle or checking whether a fact belongs in8a script, asset, or reference. The parent manual owns first-call behavior;9[operation-contract.md](../operation-contract.md) owns exact Web semantics; the10[tier index](../tier-quick-refs/SKILL.md) owns recovery routing.1112## Retained procedure support1314| Path | Owner | Use |15|---|---|---|16| `scripts/extract_page.py` | executable helper | Explicit `--tier`, `--search`, `--fallback`, and `auto_tier()` procedures. |17| `scripts/cached_get.py` | executable helper | Legacy cache helper; currently broken on its normal successful-GET path (below). |18| `assets/api-endpoints.json` | endpoint data | Academic, search, extraction, and real-time endpoint metadata. |19| `assets/css-selectors.json` | selector data | Reusable extraction selectors. |20| `assets/site-templates.json` | site data | Known-site selectors and endpoint hints. |21| `assets/extraction-pipeline.json` | pipeline data | Legacy text: not valid JSON; not a runnable pipeline or Web policy. |22| `assets/regex-patterns.json` | identifier data | DOI/arXiv/PMID/PMC/ISBN and URL patterns. |23| `assets/search-providers.json` | external provider data | Explicit vendor recipes; verify availability and authorization. |2425Scripts describe their own implementation only, not the public Web contract.26Assets are historical hints, not current authority: fixed free/unlimited/quota27claims, example contacts, package names and endpoints require upstream checking. Inspect current vendor documentation for endpoints, quotas,28dependencies, and prices; this bundle grants no install, credential, paid-use,29network, or access-control authority. Never put secrets or private paths in30examples or diagnostics.3132## Known legacy limitations3334- `cached_get.py` has no imported JSON module and its `json` argument shadows35 that name. A normal successful GET reaches `json.dump` and raises36 `AttributeError` after opening its cache file for writing; an existing file37 may already have been truncated. Cache reads also fail. Do not recommend it38 as a working cache or run it over shared/pre-existing data. Its cache key39 includes URL/method but excludes params, headers and identity. No automatic40 purge occurs: old claims about one-day/200-entry eviction were not implemented.41- Cache defaults are `/tmp/web-browsing-cache`, overridden by42 `WEB_BROWSING_CACHE_DIR`, and TTL `3600`, overridden by43 `WEB_BROWSING_CACHE_TTL` at import. `clear_cache(url)` unlinks that GET entry;44 `clear_cache()` unlinks every matching JSON file. Both require explicit45 deletion authorization; neither is a diagnostic. The CLI prints a preview.46- `assets/extraction-pipeline.json` is not valid JSON (`500/month` is an47 unquoted expression). Read it only as legacy notes, not machine configuration.48 The other parseable assets are not automatically consumed by public Web.49- `extract_page.py` is a separate Requests/browser/vendor helper, not the vetted50 public BrowserEngine. Its preview fields are truncated (including the JSON51 saved by `--json`); exit 0 means no `error` key, not complete text or useful52 content. CrossRef helper `is_oa` actually contains a citation count, not an OA53 boolean; verify OA against the real source. `--save`/`--json` overwrite paths.54- `--fallback` can advance through local tiers to **Jina**, sending the target55 URL to an external service. `auto_tier` is a heuristic; explicit `--tier`56 still may do several metadata requests. Authorize the whole selected route,57 dependencies and data disclosure before execution; no implicit chain.58 Helper diagnostics may include raw URLs/exceptions: never feed secrets or59 private URLs. Tier 3 blocks images/styles/fonts/media and returns only a60 preview, so it is not a visual-validation harness.6162These are preserved implementation limitations, not runtime fixes in the manual63reduction. For a proposed repair, use the `lingtai-issue-report` protocol and64obtain filing/implementation authority; do not silently patch installed helpers.6566## Cleanup / Footprint6768Public Web leaves complete `tmp/tool-results/web-*` artifacts; snapshots,69cursors and references are bounded process-local state. Optional helpers leave70explicit downloads/JSON outputs and the cache described above. None are71implicitly disposable: preserve live referenced results, evidence, user data,72configuration and secrets, and never blindly clear a shared cache.7374After a large retrieval session or before retirement, follow `skills-manual` →75`reference/cleanup-footprint-contract.md#shared-footprint-check-recipe`. Select76only this Agent's `tmp/tool-results/web-*` plus the exact task-created output77paths; inspect a configured helper cache separately only when authorized.78The shared recipe reports count/bytes without changing inspected files; its79optional `logs/cleanup.jsonl` append is a separate explicit write. Present a80dry-run listing what stays/goes, obtain explicit human consent, then perform81only the approved cleanup and record timestamp/tool/dry-run-or-apply/count/bytes/82path summary/approval. If consent is absent, stop at inspection.