How do I install the Scanner skill?

Run npx skillmds@latest add google-gemini/scanner in your terminal (requires Node.js), paste this page's agent-chat prompt into Claude, Cursor, or any MCP-connected agent, or download the SKILL.md file and copy it into your agent's skills directory.

What does the Scanner skill do?

Scans a website deeply, converting HTML pages to markdown, respecting robots.txt, and updating the snapshots log. It is listed under Web & Frontend on SkillMD.

Is Scanner safe to use?

This skill has not completed SkillMD's automated safety review yet. SkillMD never runs a skill's scripts for you; review the SKILL.md before installing.

Which AI agents work with Scanner?

This skill is tagged as working with Claude Code, Claude.ai, OpenAI Codex. SKILL.md is an open format, so most agents that read a skills directory can load it too.

Is Scanner free to use?

Yes. Installing skills from SkillMD is free, and the skill stays under its author's original license.

Who published Scanner?

Google Gemini (@google-gemini) published this skill. Their other Agent Skills are listed on their SkillMD profile.

Scanner by google-gemini — Agent Skill

Name: Scanner
Author: Google Gemini

File contents

Scanner Skill

Use this skill to scan and analyze all relevant pages under a target website domain to build or refresh a local customer support corpus.

Embedded Script

python skills/scanner/scripts/scan.py <URL> [--force]

Arguments

Argument	Description
`<URL>`	The start/seed URL of the website to analyze (e.g. `https://example.com`)
`--force`	Force scanning and bypass the 24-hour cache check

Features

Robots.txt Compliance: Checks robots.txt before parsing. If restricted, aborts scanning. Make sure to perform scanning only on sites that are allowed.
Domain-Locked Recursive Scanning: Only analyzes links within the same domain/subdomain to avoid leaking to other websites.
HTML to Markdown: Converts HTML structure into clean, readable Markdown text suitable for LLM document matching.
Caching & Snapshot Maintenance: Creates or updates .agents/workspace/snapshots.json with mapping and timestamp.
Corpus Directory Index: Automatically generates .agents/workspace/pages/index.md listing and explaining all files in a structured, clickable table format so agents can locate matching topics immediately.

Frequently asked questions

Scanner