agentskillexchange SiteOne Crawler Technical SEO and Site Audit
SiteOne Crawler is a real website crawler and analyzer for technical SEO, accessibility, security, and performance checks. This skill uses the upstream SiteOne Crawler project to turn large site crawls into structured diagnostics, export files, and remediation queues.
Installation
Use the upstream install or setup path that matches your environment:
- brew install janreges/tap/siteone-crawler
- git clone https://github.com/janreges/siteone-crawler.git
- cargo build --release
- cargo build --release --target x86_64-unknown-linux-musl
Requirements and caveats from upstream:
- | glibc (primary) | Requires glibc 2.39+ (Ubuntu 24.04+, Debian 13+, Fedora 40+) | Full native performance |
- Requires Rust 1.85 or later.
- | --mail-smtp-user= | SMTP user, if your SMTP server requires authentication. |
Basic usage or getting-started notes:
Now rewritten in Rust for maximum performance, minimal resource usage, and zero runtime dependencies. The transition from PHP+Swoole to Rust resulted in 25% faster execution and *30% lower memory consumption...
Run Anywhere: Single native binary for 🪟 Windows, 🍎 macOS, and 🐧 Linux (x64 & arm64). No runtime dependencies.
Deep Crawling & Analysis: Thoroughly crawl every page and asset, identify errors (404s, redirects), generate sitemaps 🗺️, and even get email summaries 📧 (watch [▶️ video example](https://www.youtube.com/wa...
Extracted from upstream docs: https://raw.githubusercontent.com/janreges/siteone-crawler/HEAD/README.md