# Ingest Link

> Use when the user wants to ingest web URLs, online PDFs, rendered web pages, standards, manuals, or articles through qt-web-extractor into the normal document-ingest workflow.

- Skill: `zimoliao/ingest-link` (Agent Skill)
- Install (CLI): `npx skillmds@latest add zimoliao/ingest-link`
- Raw SKILL.md: https://api.skillmd.com/api/skills/zimoliao/ingest-link/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Productivity
- Author: zimoliao (https://skillmd.com/u/zimoliao)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/zimoliao/ingest-link

---


# Ingest Web Links

Use this skill when the user already has one or more URLs and wants them pulled into the ScholarAIO library as document-style records.

## When to Use

Use this skill when the user wants to:

- ingest a webpage directly into the local knowledge base
- ingest an online PDF or report from a URL
- capture technical documentation, standards, manuals, or web articles as `document` items
- route rendered web content through the normal ScholarAIO ingest/index flow

Do not use this skill when:

- the task is only to search the local library; use `search`
- the task is mainly about arXiv preprints; use `arxiv`
- the user only wants broad web discovery without ingesting; use external web search first, then come back if ingestion is desired

## Core Workflow

### 1. Confirm the source shape

- If the user already provides URL(s), go straight to `scholaraio ingest-link`
- If the user only has a topic but no URL, discover URLs first through the host agent's native web search, then ingest selected results

### 2. Use the CLI entrypoint

Basic ingestion:

```bash
scholaraio ingest-link https://example.com/page
scholaraio ingest-link https://example.com/page https://example.com/report.pdf
```

Preview only:

```bash
scholaraio ingest-link https://example.com/page --dry-run
```

Ingest without rebuilding indexes immediately:

```bash
scholaraio ingest-link https://example.com/page --no-index
```

Force PDF extraction mode when the backend needs a hint:

```bash
scholaraio ingest-link https://example.com/report.pdf --pdf
```

### 3. Keep the backend model clear

- ScholarAIO does not render webpages itself in this flow
- It depends on an external `qt-web-extractor` service
- The value of that service is rendered-content extraction, not just raw HTML download
- Default HTTP endpoint: `http://127.0.0.1:8766`
- Preferred MCP endpoint: `http://127.0.0.1:8766/mcp`
- Prefer `config.yaml -> webextract.transport: mcp` plus `webextract.mcp_url` / `webextract.mcp_tool`
- HTTP fallback uses `webextract.base_url` / `webextract.api_key`
- Environment variables `WEBEXTRACT_TRANSPORT`, `WEBEXTRACT_MCP_URL`, `WEBEXTRACT_URL`, and `WEBEXTRACT_API_KEY` still work as overrides

### 4. Understand what gets stored

- The extracted page is written into a temporary document inbox
- ScholarAIO reuses the existing document ingest flow
- Final records stay in the current `document` family, not a separate `webdocument` type
- Provenance fields such as `source_url`, `source_type`, and `extraction_method` are preserved in `meta.json`
- If the extractor returns rendered HTML, ScholarAIO discovers image references, downloads reachable images into a local `images/` directory during ingest, and rewrites Markdown links to local relative paths.
- If an image download fails because of access control, SSL, 404, or similar issues, ScholarAIO keeps the original external URL in the Markdown instead of breaking the content.

## Practical Heuristics

- Prefer `--no-index` when ingesting many links and you plan to rebuild once at the end
- Let the backend auto-detect normal web pages and PDF URLs first; prefer `--pdf` only when detection seems unreliable
- If the backend is unavailable, report that clearly instead of pretending ScholarAIO can fetch/render the page alone

## Output Style

- Make it clear that the content came from a URL rather than a local file
- Mention the source URL in summaries when it matters
- If ingestion fails, surface the backend/service reason directly