Web Resource Checker

Web Resource Checker

Validates essential web resource files for SEO, security, and LLM accessibility compliance.

Supported Files

Lookup Workflow

1. Identify the query type:

   • File specification lookup (e.g., "sitemap.xml format", "security.txt fields")
   • Validation request (e.g., "check my sitemap", "validate robots.txt")

2. For specification lookup:

   • Read web-resources-index.json
   • Return summary, requirements, and official reference

3. For validation:

   • Run the appropriate validator script
   • Return issues found with severity and fix suggestions

Usage

Validate All Resources

# Check local directory
cd ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker && npx web-resource-checker ./public

# Check live site
cd ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker && npx web-resource-checker https://example.com

# JSON output
cd ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker && npx web-resource-checker https://example.com --json

# Check specific files only
cd ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker && npx web-resource-checker https://example.com --only=sitemap,robots

# With custom timeout (in milliseconds)
cd ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker && npx web-resource-checker https://example.com --timeout=30000

Lookup Specifications

# Get sitemap.xml specification
cat ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker/web-resources-index.json | jq '.files["sitemap.xml"]'

# Get security.txt required fields
cat ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker/web-resources-index.json | jq '.files["security.txt"].requiredFields'

# Get llms.txt example
cat ${CLAUDE_PLUGIN_ROOT}/skills/web-resource-checker/web-resources-index.json | jq -r '.files["llms.txt"].example'

Response Format

For Specification Lookup

### [File Name]

**Summary**: [1-2 sentence explanation]

**Required Elements**:
- [Element 1]
- [Element 2]

**Common Issues**:
- [Issue description]

**Official Specification**: [URL]

For Validation Results

### [File Name] - [Status]

**Issues Found**: [count]

1. **[Issue Title]** ([severity])
   - Problem: [description]
   - Fix: [suggestion]

2. ...

Quick Reference

sitemap.xml

• Location: /sitemap.xml
• Max URLs: 50,000 per file
• Max Size: 50MB uncompressed
• Required: <urlset>, <url>, <loc>
• Recommended: <lastmod>, <changefreq>, <priority>

robots.txt

• Location: /robots.txt (site root only)
• Common Directives: User-agent, Disallow, Allow, Sitemap
• Case: Directives are case-insensitive
• Tip: Always include Sitemap: directive

llms.txt (llmstxt.org)

• Location: /llms.txt
• Format: Markdown
• Required: H1 title (# Site Name)
• Recommended: Blockquote summary, H2 sections with links

llms-full.txt

• Location: /llms-full.txt
• Purpose: Complete documentation for LLM context
• Size: Should fit in ~100k tokens

security.txt (RFC 9116)

• Location: /.well-known/security.txt (preferred) or /security.txt
• Required Fields: Contact, Expires
• Expires Format: ISO 8601 (e.g., 2025-12-31T23:59:59.000Z)
• HTTPS: Must be served over HTTPS

Common Patterns

Complete sitemap.xml

<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/</loc>
    <lastmod>2024-01-15</lastmod>
    <changefreq>daily</changefreq>
    <priority>1.0</priority>
  </url>
  <url>
    <loc>https://example.com/about</loc>
    <lastmod>2024-01-10</lastmod>
    <changefreq>monthly</changefreq>
    <priority>0.8</priority>
  </url>
</urlset>

Complete robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /private/
Disallow: /api/

User-agent: GPTBot
Disallow: /

Sitemap: https://example.com/sitemap.xml

Complete llms.txt

# Example Site

> A comprehensive platform for developers to learn and build modern applications.

## Documentation

- [Getting Started](https://example.com/docs/start): Quick start guide for new users
- [API Reference](https://example.com/docs/api): Complete API documentation
- [Tutorials](https://example.com/docs/tutorials): Step-by-step guides

## Optional

- [Blog](https://example.com/blog): Latest updates and articles
- [Changelog](https://example.com/changelog): Version history

Complete security.txt

Contact: mailto:security@example.com
Contact: https://example.com/security-contact
Expires: 2025-12-31T23:59:59.000Z
Encryption: https://example.com/.well-known/pgp-key.txt
Acknowledgments: https://example.com/security/thanks
Preferred-Languages: en, ja
Canonical: https://example.com/.well-known/security.txt
Policy: https://example.com/security-policy
Hiring: https://example.com/careers/security

External Resources

• Google Search Central - Sitemaps
• Google Search Central - Robots.txt
• llmstxt.org - Official Specification
• securitytxt.org - Generator
• CISA - security.txt Recommendation