# Web Research

> Web research automation — search the web, scrape readable page content, and summarize text. Use when the user needs to search the web, look something up online, find information on the internet, scrape a web page, extract article text from a URL, read a webpage's content, or summarize long text or an article. Triggers include "search the web", "cari di internet", "look this up", "google this", "scrape this page", "scrape URL", "extract text from", "read this article", "get content from this link", "summarize this", "ringkas", "summarize this article", "research this topic", "find info about", or any task requiring web search, page scraping, or text summarization. No API key required — uses only the Python standard library.

- Skill: `lazuardytech/web-research` (Agent Skill, multi-file: 7 files)
- Install (CLI): `npx skillmds@latest add lazuardytech/web-research`
- Raw SKILL.md: https://api.skillmd.com/api/skills/lazuardytech/web-research/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Research & Search
- License: MIT
- Author: lazuardytech (https://skillmd.com/u/lazuardytech)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/lazuardytech/web-research

---


# web-research

Web search, page scraping, and text summarization using only the Python
standard library. No API keys, no third-party packages.

Run CLIs from this skill's directory (the folder that contains this `SKILL.md`):
`uv run scripts/<name>.py` or `python3 scripts/<name>.py`. Do not prefix
`skills/web-research/` — that path is only valid from this monorepo's root.

## Available Scripts

### Search the web

```bash
uv run scripts/search.py --query "climate policy 2024" --limit 10
```

Searches via the DuckDuckGo HTML endpoint. Outputs JSON:
`{"query": ..., "count": N, "results": [{"title", "url", "snippet"}]}`.
Exit `2` if no results.

### Scrape a page

```bash
uv run scripts/scrape.py --url "https://example.com/article" --max-chars 5000
```

Fetches and extracts the readable title, meta description, and text. Outputs
JSON: `{"url", "title", "description", "text"}`. Exit `2` if no text extracted.

### Summarize text

```bash
uv run scripts/scrape.py --url "https://example.com" | \
  uv run scripts/summarize.py --sentences 5
```

Reads plain text OR JSON with a `text` field (e.g. `scrape.py` output) on
stdin. Produces an extractive, frequency-ranked summary. Outputs JSON:
`{"sentences": N, "summary": [...]}`. Works offline once text is on stdin.

## Pipeline

Chain the scripts: `search.py` → pick a URL → `scrape.py` → `summarize.py`.

## Output & Exit Codes

- All scripts emit JSON to stdout; diagnostics to stderr.
- Exit `0` success, `1` error, `2` not found.

## Notes

- Search uses DuckDuckGo's HTML endpoint; heavy use may be rate-limited.
- Summarization is extractive (no LLM), so it needs no API key and runs offline.