Get Web Page Raw

Get Web Page (Raw)

When to use

Use this skill when:

• The user provides a URL to capture as source material
• You need to store a web page for later distillation
• You're building up raw materials for the distillation pipeline

Keywords: capture, web page, URL, fetch, raw

Inputs

Required:

• url (string): The URL of the web page to capture

Optional:

• title_hint (string): A human-provided hint for the title if extraction fails

Outputs

This skill produces:

1. A new file in raw/ with the naming pattern: YYYYMMDD-HHMMSSZ--<slug>.md
2. Metadata returned to the agent:
   • raw_path: full path to the created file
   • title: extracted or inferred title
   • canonical_url: canonical URL if found, otherwise the input URL
   • captured_at: ISO timestamp (UTC)

Procedure

1. Generate timestamp and slug

• Get current UTC timestamp: date -u +"%Y%m%d-%H%M%SZ"
• Create a slug from the page title or URL path:
  • Use the main topic/title in lowercase
  • Replace spaces and special chars with hyphens
  • Keep it concise (typically 3-8 words)
  • Example: patterns-for-ai-assisted-software-development

2. Fetch the web page content

Primary method: fetch_webpage tool

• Call fetch_webpage with the URL and a descriptive query about the content
• This extracts main content, removes navigation/ads, and converts to clean Markdown
• Captures metadata like title and author when available

Alternative methods (when needed):

• Playwright MCP: For complex interactive pages, JavaScript-heavy sites, or when you need precise browser control
• curl + HTML parsing: For simple pages when fetch_webpage isn't available, though requires manual Markdown conversion
• Browser tools: Built-in browser navigation and snapshot capabilities

Note: Most public articles and blog posts work well with fetch_webpage

3. Build front matter

Create YAML front matter with required and available fields:

---
title: "<extracted from page>"
source_url: "<original URL provided>"
captured_at: "<ISO timestamp from step 1>"
capture_type: web_page
capture_tool: fetch_webpage  # or browser if used
raw_format: markdown
status: captured
author: "<if clearly visible>"  # optional
published_at: "<YYYY-MM-DD if available>"  # optional
tags: ["keyword1", "keyword2"]  # optional, 3-5 relevant tags
---

Tips:

• Use the page's actual title, not a description
• Convert published_at to simple date format (YYYY-MM-DD)
• Tags should be broad topics, not exhaustive keywords

4. Write the file

• Filename: <timestamp>--<slug>.md (e.g., 20260102-095107Z--patterns-for-ai-assisted-software-development.md)
• Full path: raw/<filename>
• Content: front matter + blank line + extracted content
• Use create_file tool with the full content

5. Confirm to user

Provide a brief confirmation:

• Link to the created file using relative path
• One-sentence summary of what the article covers (2-3 key topics)

Examples

Example 1: Dev.to article capture

Input:

url: "https://dev.to/javatarz/patterns-for-ai-assisted-software-development-4ga2"

Actions:

1. Generate timestamp: 20260102-095107Z
2. Fetch with fetch_webpage tool
3. Extract title: "Patterns for AI assisted software development"
4. Create slug: patterns-for-ai-assisted-software-development
5. Write to: raw/20260102-095107Z--patterns-for-ai-assisted-software-development.md

Output:

---
title: "Patterns for AI assisted software development"
source_url: "https://dev.to/javatarz/patterns-for-ai-assisted-software-development-4ga2"
captured_at: "2026-01-02T09:51:07Z"
capture_type: web_page
capture_tool: fetch_webpage
raw_format: markdown
status: captured
author: "Karun Japhet"
published_at: "2025-12-27"
tags: ["ai", "programming", "productivity", "patterns"]
---

Edge cases

fetch_webpage fails:

• Check if authentication is required
• Try browser-based extraction if needed
• Inform user if the page cannot be accessed

Missing metadata:

• If no title is extracted, use the URL path or ask user for a title hint
• If no author/date visible, omit those fields
• Minimal front matter is acceptable

Paywalled or restricted content:

• Inform user about the restriction
• Capture what's visible if useful (preview text)
• Consider asking if they have alternative access

References

• Parent workflow: docs/distillation/distillation-pipeline.md
• Front matter schema: see "Raw file front matter" in distillation-pipeline.md
• Next step after capture: use make-distilled skill to process this raw file