Source Normalization

Source Normalization

Purpose

Convert diverse source formats into standardized citation entries:

• Normalize file paths, URLs, and pasted text
• Generate stable source IDs (hash-based)
• Calculate checksums for change detection
• Create consistent metadata structure
• Deduplicate by content or URL
• Maintain citations/sources.json registry

When to Use This Skill

Activate automatically when:

• content-run workflow processes input files/URLs/text
• research-processing workflow adds external sources
• User provides mixed input formats (files + URLs + text)
• Any workflow needs citation tracking
• Building content with source attribution

Source Input Types

1. File Inputs

Format: Absolute or relative paths, glob patterns

Examples:

/Users/jay/llm/datasets/meetings/Customers/PrettyBoy/2025/10-15_sales_discovery-call.md
datasets/learning/sources/seasonality/**/*.md
~/Workspace/llm/datasets/research/competitive-analysis/klaviyo-pricing.md

Normalization:

• Expand globs to individual files
• Convert relative paths to absolute
• Verify file existence
• Calculate checksum
• Extract metadata from frontmatter (if present)

2. URL Inputs

Format: HTTP/HTTPS URLs

Examples:

https://www.klaviyo.com/blog/segment-examples
https://example.com/analyst-report.pdf
https://github.com/company/repo/README.md

Normalization:

• Fetch content (if --web enabled)
• Save to inputs/ directory as markdown
• Extract title from HTML or first H1
• Calculate checksum of fetched content
• Store original URL in metadata

3. Text Inputs

Format: Pasted text blocks, inline context

Examples:

TEXT:
>>>
Add a counterpoint on over-segmentation and a simple prompt example.
User feedback from Slack: "We need better export options for campaign data."
<<<

Normalization:

• Save to inputs/context_{timestamp}.md
• Generate title from first sentence or timestamp
• Calculate checksum
• Mark kind as "note"

Normalization Process

1. Detect Source Type

For each input:

If starts with "http://" or "https://":
  → Type: URL
Else if file path (contains "/" or "\" or exists as file):
  → Type: FILE
Else if in TEXT block or pasted content:
  → Type: TEXT

2. Expand and Validate

FILE inputs:

1. Expand glob patterns to file list
2. Convert relative paths to absolute
3. Verify each file exists (stat check)
4. If missing: Log error, skip

URL inputs:

1. Validate URL format (regex)
2. If `--web` enabled:
   - Fetch content using WebFetch
   - Save to inputs/fetched_{slug}_{timestamp}.md
   - Extract title from <title> or first H1
3. If `--web` not enabled:
   - Create placeholder entry (URL stored, content not fetched)

TEXT inputs:

1. Write to inputs/context_{timestamp}.md
2. Generate title from first 5-10 words or "Context {timestamp}"

3. Calculate Checksum

For all source types:

# Read file content (or fetched content for URLs)
content=$(cat /path/to/source.md)

# Calculate SHA-256 checksum
checksum=$(echo "$content" | sha256sum | awk '{print "sha256:"$1}')

Why checksums matter:

• Detect file modifications
• Enable change-based sync (Mochi flashcards)
• Verify content integrity
• Support caching and deduplication

4. Generate Stable ID

ID format: src_{first8chars_of_checksum}

Example:

Checksum: sha256:a7f3b2c19d4e5f6g...
ID: src_a7f3b2c1

Why stable IDs:

• Consistent reference across workflows
• URL-safe identifiers
• Content-based (same content = same ID)
• Short and readable

5. Extract Metadata

From file frontmatter (if YAML present):

title: "Source Title"
author: "Author Name"
published_date: "YYYY-MM-DD"
url: "https://original-url.com" (if this file was fetched)
topic: "strategic-category"
tags: ["keyword1", "keyword2"]

From HTML (for fetched URLs):

<title>Page Title</title>
→ Extract as title

<meta name="author" content="Author Name">
→ Extract as author

<meta name="description" content="Description">
→ Extract as summary

Fallback values:

• title: Filename or first H1 or "Untitled Source"
• author: null
• published_date: null (or fetch date for URLs)

6. Create Citation Entry

Standard schema:

{
  "id": "src_a7f3b2c1",
  "title": "Source Title",
  "kind": "file" | "url" | "note",
  "path": "/absolute/path/to/source.md",  // for kind=file or saved URLs
  "url": "https://...",  // for kind=url (original URL)
  "checksum": "sha256:a7f3b2c1...",
  "added_utc": "2025-10-21T14:30:00Z",
  "author": "Author Name" (optional),
  "published_date": "YYYY-MM-DD" (optional),
  "topic": "strategic-category" (optional),
  "tags": ["keyword1", "keyword2"] (optional)
}

7. Deduplicate

Deduplication strategies:

By checksum:

If sources.json already contains entry with same checksum:
  → Skip (identical content already registered)
  → Log: "Source already exists: {existing_id}"

By URL (for URL inputs):

If sources.json already contains entry with same URL:
  → Check if content changed (compare checksums)
  → If changed: Update entry with new checksum and timestamp
  → If unchanged: Skip

By path (for FILE inputs):

If sources.json already contains entry with same absolute path:
  → Check if content changed (compare checksums)
  → If changed: Update entry with new checksum
  → If unchanged: Skip

8. Write to citations/sources.json

File structure:

{
  "sources": [
    {
      "id": "src_a7f3b2c1",
      "title": "Source 1",
      "kind": "file",
      "path": "/path/to/source1.md",
      "checksum": "sha256:a7f3b2c1...",
      "added_utc": "2025-10-21T14:30:00Z"
    },
    {
      "id": "src_d4e5f6g7",
      "title": "Source 2",
      "kind": "url",
      "path": "/path/to/inputs/fetched_source2.md",
      "url": "https://example.com/source2",
      "checksum": "sha256:d4e5f6g7...",
      "added_utc": "2025-10-21T14:35:00Z"
    }
  ],
  "version": "1.0",
  "last_updated": "2025-10-21T14:35:00Z"
}

Write operation:

1. Read existing citations/sources.json (if exists)
2. Append new entries (or update existing)
3. Sort by added_utc (descending)
4. Write back to file with pretty formatting (indent: 2)

Output Structure

Created files:

content/{date}_{type}_{slug}/
├── inputs/
│   ├── {original_filename}.md  (copied from FILE input)
│   ├── fetched_{slug}_{timestamp}.md  (from URL input)
│   └── context_{timestamp}.md  (from TEXT input)
└── citations/
    └── sources.json  (normalized citation registry)

Integration with Workflows

Content Pipeline Integration

Invoked by:

• content-run workflow (after intent gathering, before brief creation)
• content-intent-gathering skill (processes FILES/URLS/TEXT sections)

Inputs:

• FILES: Array of file paths (may include globs)
• URLS: Array of URLs
• TEXT: Pasted text blocks
• --web flag: Whether to fetch URLs

Outputs:

• Normalized files in inputs/ directory
• citations/sources.json with all source entries
• Ready for citation in brief/outline/draft

Research Processing Integration

Invoked by:

• research-processing workflow

Inputs:

• External source URL or file path
• Topic category for organization

Outputs:

• Normalized source file in datasets/research/{topic}/
• Citation entry in sources.json
• Checksum for integrity validation

Quality Gate Integration

Used by:

• citation-compliance (reads sources.json for verification)
• source-integrity (validates checksums in sources.json)
• link-verification (uses URL entries for validation)

Success Criteria

Source normalization complete when:

• All input sources processed (files, URLs, text)
• Stable IDs generated for each source
• Checksums calculated
• Metadata extracted (or sensible defaults applied)
• Deduplication applied (no duplicate entries)
• citations/sources.json created/updated
• All source files accessible in inputs/ directory

Common Mistakes

Related Skills

• source-integrity: Validates checksums and metadata completeness
• citation-compliance: Uses sources.json for verification
• link-verification: Validates URL accessibility
• content-intent-gathering: Provides source inputs for normalization
• research-processing: Uses normalization for external sources

Anti-Rationalization Blocks

Common excuses that are explicitly rejected: