Blog Scraper

Blog Scraper Skill

Overview

This skill fetches blog articles from tech-lab.sios.jp/archives/*, compresses the HTML content by removing unnecessary attributes and whitespace, and saves the result to the doc/ directory with metadata.

When to Use

• User requests to fetch a specific blog article
• User wants to update existing cached articles
• User needs to scrape multiple articles for analysis or documentation

Usage

Single Article

URL=https://tech-lab.sios.jp/archives/[article-id] npm run scraper

Example:

URL=https://tech-lab.sios.jp/archives/48397 npm run scraper

Multiple Articles

For multiple articles, run the command sequentially for each URL.

Output

The scraper will:

1. Fetch and parse the HTML from the specified URL
2. Extract content using the CSS selector section.entry-content
3. Compress by removing:
   • Scripts, styles, and noscript tags
   • Class, ID, and style attributes
   • Whitespace between tags
4. Preserve:
   • Image alt text as [画像: alt]
   • Image src URLs
   • Link href attributes
5. Add metadata as HTML comment:
   • OGP title
   • Source URL
   • OGP image URL
   • Extraction timestamp
6. Save to docs/data/tech-lab-sios-jp-archives-[id].html
7. Report compression statistics:
   • Token count reduction (estimated for Claude)
   • Compression ratio percentages
   • File size

Cache Behavior

• If the target HTML file already exists in docs/data/, the scraper skips fetching and reports the existing file size
• To re-fetch, delete the existing HTML file first

Token Estimation

The scraper estimates Claude token usage for Japanese content:

• Hiragana/Katakana: ~1.5 chars/token
• Kanji: ~1 char/token
• ASCII: ~4 chars/token
• Other: ~2 chars/token

Typical compression achieves 60-85% token reduction.

Implementation Details

See application/tools/scraper.ts for the TypeScript implementation using:

• node-fetch for HTTP requests
• cheerio for HTML parsing
• OGP metadata extraction
• Custom token estimation for Japanese text

Permissions Required

This skill requires the following permissions in .claude/settings.local.json:

{
  "permissions": {
    "allow": [
      "Bash(npm run scraper:*)",
      "Bash(URL=:*)"
    ]
  }
}

Note: The Bash(URL=:*) permission uses prefix matching to allow any URL environment variable pattern. This is a broad permission - consider restricting to specific domains if needed for security.