Fumadocs Article Importer

Fumadocs Article Importer

Automate importing external articles into a Fumadocs project with tri-language support (English, Chinese, French), auto-classification, and proper MDX formatting.

Prerequisites

Before using this skill, verify:

• Fumadocs project is initialized in the current directory
• Jina MCP is configured for article fetching (highly recommended)
  • Repository: https://github.com/jina-ai/MCP
  • Provides 15 tools for content extraction, search, and processing
  • See "MCP Configuration" section below for setup
  • Alternative: Jina API access (via curl)
• Translator Skill is available for professional translation
  • Located in .claude/skills/translator/
  • Provides professional translation using Claude's native capabilities
  • Automatically activated when translation is needed
  • No external dependencies or configuration required
• curl is installed for image downloads
• Write access to content/docs/ and public/images/ directories

⚠️ CRITICAL REQUIREMENTS

1. Must Use withAllImages: true Parameter

For image processing to work, you MUST use withAllImages: true in Step 2:

Tool: read_url
Parameters:
  - url: {article_url}
  - withAllImages: true  // ← MANDATORY for image extraction

Why this matters:

• Without withAllImages: true: Jina returns text only, no images
• With withAllImages: true: Jina returns text + images array
• If you skip this parameter, ALL image processing will be silently skipped
• See Step 2, Sub-step 4 for mandatory validation check

What happens if you forget:

// WRONG - Won't extract images
const response = await mcp.read_url(url);  // Missing withAllImages!
console.log(response.images);  // ❌ undefined

// CORRECT - Will extract images
const response = await mcp.read_url(url, { withAllImages: true });
console.log(response.images);  // ✅ [img1, img2, ...]

2. Image Storage Strategy (Choose One)

You have two options for handling images. Choose before starting Step 4:

Option A: Download Images to Local (Default)

What: Download image files to public/images/docs/{slug}/

When to use:

• Source website doesn't support CORS (see CORS testing below)
• You want offline availability (images work without internet)
• You want control over image versions (won't change unexpectedly)
• Source images might be deleted or moved

Pros:

• ✅ Works 100% of the time (no CORS issues)
• ✅ Images always available (offline)
• ✅ Full control over image files
• ✅ Faster loading (no external HTTP requests)

Cons:

• ⚠️ Uses local storage space (adds 10KB-500KB per image)
• ⚠️ Increases import time (needs to download each image)
• ⚠️ Adds complexity (need to manage local files)

Example in MDX:

![MCP Diagram](/images/docs/skills-explained/mcp-architecture.png)

Option B: Use External Image URLs (No Download)

What: Keep original URLs in the article (don't download)

When to use:

• Source website supports CORS (tested with 200 + CORS headers)
• You want to save storage space
• You want faster import process
• You're okay with external dependencies
• Source images are stable and unlikely to be deleted

How to test if external URLs work:

# Test 1: Does the URL return 200?
curl -I "https://example.com/image.png"
# Expected: HTTP/2 200

# Test 2: Does it support CORS?
curl -I "https://example.com/image.png" | grep -i "access-control"
# Expected: access-control-allow-origin: *
# Or: access-control-allow-origin: https://your-domain.com

Pros:

• ✅ No storage overhead (no local files)
• ✅ Faster import (skip download step)
• ✅ Simpler process (no file management)
• ✅ Images auto-update if source changes

Cons:

• ❌ Requires CORS support (will break in browser if not)
• ❌ Requires internet connection (offline doesn't work)
• ❌ Source might delete/move images (breaks your article)
• ❌ Slower page loads (external HTTP requests)

Example in MDX:

![MCP Diagram](https://cdn.example.com/mcp-architecture.png)

Real-World Example: Claude.com Images

Test results for https://claude.com/blog/skills-explained:

# Test MCP diagram image
curl -I "https://cdn.prod.website-files.com/68a44d4040f98a4adf2207b6/69141f0993d68ff4c536f316_619a5262.png"

# Response includes:
# HTTP/2 200
# access-control-allow-origin: *

Result: ✅ Claude.com images support CORS (can use external URLs)

Decision for Claude.com articles: Use Option B (External URLs) - no download needed

Decision for unknown sources: Use Option A (Download) - safer

3. Validation Check Added in v2.0

Step 2 now includes a mandatory validation (Sub-step 4) that:

• Checks if response.images exists
• Throws clear error if missing
• Prevents silent failures
• Provides exact fix instructions

Always verify images were extracted before proceeding!

MCP Configuration

This skill works best with:

1. Jina MCP - For article fetching and content extraction
2. Translator Skill - For professional translation (built-in, no configuration needed)

Jina MCP Setup (Article Fetching)

Public Jina MCP Server (Recommended):

Add to your Claude configuration:

{
  "mcpServers": {
    "jina": {
      "url": "https://mcp.jina.ai/sse",
      "headers": {
        "Authorization": "Bearer ${JINA_API_KEY}"  // Optional, for higher rate limits
      }
    }
  }
}

Note: Works without API key but has rate limits. For production use, get a free API key at https://jina.ai

Self-Hosted Jina MCP (Optional):

git clone https://github.com/jina-ai/MCP.git
cd MCP && npm install && npm run start

Available Tools:

• read_url - Convert webpage to markdown ✨ (primary tool)
• guess_datetime_url - Get publication date
• search_web, search_arxiv, search_images - Search capabilities
• sort_by_relevance, deduplicate_strings - Content processing

Translation Setup

Translation Strategy: This skill uses Claude's native translation capabilities through the translator skill.

The translator skill provides:

• Professional-grade translation quality
• Preservation of Markdown formatting and code blocks
• Consistent technical terminology handling
• Language-specific best practices (zh, fr, ko, en)
• No external dependencies or API keys required

How it works: When this skill needs to translate content, it will automatically trigger Claude to use the translator skill. You don't need to configure anything - Claude will compose the two skills automatically based on the task requirements.

Workflow

Step 1: Get Article Information

Ask the user for the following information:

1. "What is the URL of the article you want to import?"
2. "What languages should I translate to? (Press Enter for default: en, zh, fr)"
3. "How should I handle images? (Press Enter for default: auto)\n - auto: Check CORS and use external URLs if possible, else download\n - external: Always use original image URLs (no download)\n - download: Always download to local storage"

Step 2: Download Article Content

Using Jina MCP (Recommended - best integration with Claude):

1. Fetch article content with images (RECOMMENDED - enables smart image filtering):

   Tool: read_url
   Parameters:
     - url: {article_url}
     - withAllImages: true  ← ADD THIS
   
   Returns:
     - content: Markdown-formatted article content
     - images: Array of image objects with URLs and metadata
     - title, description, etc.
   

   Why this matters: Returns structured image data instead of parsing markdown. You can access response.images directly.

2. Get publication date (optional but recommended):

   Tool: guess_datetime_url
   Parameters:
     - url: {article_url}
   
   Returns: Detected publication and update dates
   

3. Extract metadata from the fetched content:

   • Title (from markdown H1 or metadata)
   • Author (if available in content)
   • Publication date (from guess_datetime_url or content)
   • Main content (body text)
   • All image URLs (from response.images array if withAllImages=true, else extract from markdown)
   • Detect YouTube videos (search for youtube.com/embed, youtu.be, youtube.com/watch URLs)

4. ⚠️ CRITICAL VALIDATION - Check for images (DO NOT SKIP):

   // VERIFICATION STEP - Must check before proceeding!
   
   // Check if withAllImages parameter was actually used
   if (!response.images) {
     console.error("❌ CRITICAL ERROR: response.images is undefined!");
     console.error("→ This means 'withAllImages: true' parameter was NOT passed to read_url");
     console.error("→ Image processing will be completely skipped!");
   
     // STOP here - do not proceed without image data
     throw new Error(
       `FAILED: Cannot extract images from ${article_url}\n` +
       `Cause: withAllImages parameter missing in read_url call\n` +
       `Solution: Re-run with correct parameter: { withAllImages: true }`
     );
   }
   
   // Validate images array
   if (response.images.length === 0) {
     console.warn("⚠️ WARNING: response.images array is empty!");
     console.warn("→ The article may have no images, OR extraction failed");
   
     // Ask user to confirm if this is expected
     const hasImages = confirm("Does this article have images you want to download?");
     if (hasImages) {
       throw new Error(
         `FAILED: Expected images but found none. Retry with withAllImages: true`
       );
     }
   }
   
   // SUCCESS - Log image count
   console.log(`✅ SUCCESS: Found ${response.images.length} images in article`);
   console.log(`→ Ready to proceed to image filtering (Step 3.5)`);
   

Why this validation is critical:

• Many articles contain 15-20 images, but only 1-3 are actual content
• Without withAllImages: true, you get TEXT ONLY (no images array)
• If you skip this check, you'll never know images were missed until it's too late
• This validation FORCE STOPS execution if images are missing unexpectedly

Real-world consequence of skipping this check:

// WRONG - Skipping validation:
const response = await mcp.read_url(url); // Forgot withAllImages
const images = response.images; // ❌ undefined
heuristicFilter(images); // Returns empty array (undefined becomes [])
console.log("Found 0 images"); // User thinks article has no images
// Result: No images downloaded, user doesn't know they were missed

// CORRECT - With validation:
const response = await mcp.read_url(url); // Forgot withAllImages
if (!response.images) { // ✅ Validation catches the error
  throw new Error("Missing withAllImages parameter!"); // Stops execution
}
// Result: Clear error message, user knows to retry correctly

Alternative: Using Jina API directly (if MCP not available):

# Fetch article as markdown
curl "https://r.jina.ai/{article_url}"

# With custom options
curl "https://r.jina.ai/{article_url}" \
  -H "X-Return-Format: markdown" \
  -H "X-With-Generated-Alt: true"

Fallback: If neither Jina MCP nor API is available:

• Ask user to provide article content directly
• Or use web scraping with Claude's web browsing capability
• Manual copy-paste of article content

Step 2.5: Content Safety Processing (DEFENSIVE)

Critical: Apply defensive processing to prevent MDX syntax errors. This step acts as a safety net to handle unknown components and common MDX pitfalls from ANY source, not just Anthropic.

Why this matters: Articles come from diverse sources (Anthropic, GitHub, Medium, personal blogs, etc.), each with different component libraries and Markdown flavors. Instead of crashing on unknown syntax, we safely degrade content while preserving readability.

Processing Pipeline:

// Safety processor that handles content from any source
const safetyProcessor = {
  // Phase 1: Handle unknown/dangerous JSX components
  handleUnknownComponents(content: string): string {
    // Known Fumadocs components (whitelist - safe to keep)
    const fumadocsComponents = [
      'Callout', 'Cards', 'Card', 'Tabs', 'Tab', 'Steps', 'Step',
      'Files', 'Folder', 'File', 'Accordion', 'ImageZoom'
    ];

    // Pattern 1: Handle closed components <Component>...</Component>
    content = content.replace(
      /<([A-Z][a-zA-Z]*)[^>]*>([\s\S]*?)<\/\1>/g,
      (match, componentName, innerContent) => {
        if (fumadocsComponents.includes(componentName)) {
          return match; // Keep known components
        }

        // Unknown component: degrade to plain text with comment
        console.warn(`⚠️ Unknown component <${componentName}>, degrading to plain text`);
        return `<!-- Original: <${componentName}> -->\n${innerContent}\n<!-- End: ${componentName} -->`;
      }
    );

    // Pattern 2: Handle self-closing components <Component />
    content = content.replace(
      /<([A-Z][a-zA-Z]*)[^\/]*\/>/g,
      (match, componentName) => {
        if (fumadocsComponents.includes(componentName)) {
          return match; // Keep known components
        }

        console.warn(`⚠️ Unknown self-closing component <${componentName}/>, removing`);
        return `<!-- Removed: <${componentName}/> -->`;
      }
    );

    return content;
  },

  // Phase 2: Fix common MDX pitfalls that break parsing
  fixMDXPitfalls(content: string): string {
    // Pitfall 1: <number pattern (e.g., "<5k tokens") breaks MDX
    // Replace with HTML entity or rephrase
    content = content.replace(
      /<(\d+)/g,
      (match, num) => {
        console.warn(`⚠️ Fixed <${num} pattern (breaks MDX)`);
        return `&lt;${num}`;
      }
    );

    // Pitfall 2: Common HTML-like tags in text
    const dangerousTags = ['script', 'div', 'span', 'p', 'a', 'img'];
    dangerousTags.forEach(tag => {
      content = content.replace(
        new RegExp(`<(${tag})\\b`, 'gi'),
        (match) => {
          console.warn(`⚠️ Fixed <${tag}> pattern in text`);
          return match.replace('<', '&lt;');
        }
      );
    });

    // Pitfall 3: Bold formatting without space (non-Latin languages)
    // Wrong: **粗体：**文字 → Right: **粗体：** 文字
    content = content.replace(
      /\*\*([^*]+)\*\*([^ \n*-])/g,
      '**$1** $2'
    );

    // Pitfall 4: Unclosed JSX tags (basic check)
    const tags = content.match(/<\/[a-zA-Z]+>/g);
    if (tags) {
      tags.forEach(closingTag => {
        const tagName = closingTag.replace('</', '').replace('>', '');
        const openings = (content.match(new RegExp(`<${tagName}[^>]*>`, 'g')) || []).length;
        const closings = (content.match(new RegExp(`<\/${tagName}>`, 'g')) || []).length;

        if (openings !== closings) {
          console.error(`❌ Mismatched <${tagName}> tags: ${openings} openings, ${closings} closings`);
        }
      });
    }

    return content;
  },

  // Phase 3: Auto-inject missing imports for known components
  injectImports(content: string): string {
    const usedComponents = new Set<string>();

    // Detect Fumadocs components
    const fumadocsComponents = {
      'Callout': { import: "import { Callout } from 'fumadocs-ui/components/callout';" },
      'Cards': { import: "import { Cards, Card } from 'fumadocs-ui/components/card';" },
      'Card': { import: "import { Cards, Card } from 'fumadocs-ui/components/card';" },
      'Tabs': { import: "import { Tabs, Tab } from 'fumadocs-ui/components/tabs';" },
      'Tab': { import: "import { Tabs, Tab } from 'fumadocs-ui/components/tabs';" },
      'Steps': { import: "import { Steps, Step } from 'fumadocs-ui/components/steps';" },
      'Step': { import: "import { Steps, Step } from 'fumadocs-ui/components/steps';" },
      'Files': { import: "import { Files, Folder, File } from 'fumadocs-ui/components/files';" },
      'Folder': { import: "import { Files, Folder, File } from 'fumadocs-ui/components/files';" },
      'File': { import: "import { Files, Folder, File } from 'fumadocs-ui/components/files';" },
      'Accordion': { import: "import { Accordion, Accordions } from 'fumadocs-ui/components/accordion';" },
      'ImageZoom': { import: "import { ImageZoom } from 'fumadocs-ui/components/image-zoom';" }
    };

    // Check which components are used
    Object.keys(fumadocsComponents).forEach(comp => {
      const pattern = new RegExp(`<${comp}\\b`, 'g');
      if (pattern.test(content)) {
        usedComponents.add(fumadocsComponents[comp].import);
      }
    });

    if (usedComponents.size === 0) return content;

    // Check if imports already exist
    const existingImports = content.includes('from 'fumadocs-ui/components');
    if (existingImports) {
      console.log('✅ Fumadocs imports already present');
      return content;
    }

    // Inject imports after frontmatter
    console.log(`📦 Injecting ${usedComponents.size} import statements`);
    const importBlock = Array.from(usedComponents).join('\n') + '\n\n';

    return content.replace(
      /(---\n\n)/,
      `$1${importBlock}`
    );
  }
};

// Main safety processing function
function processContentSafely(content: string, sourceUrl: string): { content: string, warnings: string[] } {
  console.log(`🔒 Processing content safely from: ${sourceUrl}`);

  const warnings: string[] = [];

  try {
    // Step 1: Handle unknown components
    content = safetyProcessor.handleUnknownComponents(content);

    // Step 2: Fix MDX pitfalls
    content = safetyProcessor.fixMDXPitfalls(content);

    // Step 3: Inject imports
    content = safetyProcessor.injectImports(content);

    console.log('✅ Content safety processing complete');
  } catch (error) {
    console.error('❌ Safety processing failed:', error);
    warnings.push(`Safety processing error: ${error.message}`);
  }

  return { content, warnings };
}

Execution in Workflow:

Call this function immediately after Step 2 (content extraction) and before Step 3 (slug generation):

// In the main workflow:
const rawContent = response.content; // From Step 2
const { content: safeContent, warnings } = processContentSafely(rawContent, articleUrl);

// Store warnings for the summary report
const processingWarnings = warnings;

// Continue with safeContent for all subsequent steps

Why This Position Matters:

• ✅ Runs BEFORE AI concept extraction (Step 3.6) → prevents AI from analyzing broken syntax
• ✅ Runs BEFORE translation (Step 6) → prevents translating component names
• ✅ Runs BEFORE cross-reference insertion (Step 3.7) → ensures clean content for link insertion
• ✅ Runs BEFORE MDX generation (Step 7) → prevents syntax errors in final file

Key Design Principles:

1. Defensive, Not Prescriptive: We don't try to perfectly convert every component. Unknown components are safely degraded rather than causing crashes.

2. Source-Agnostic: Works for ANY source (Anthropic, GitHub, Medium, personal blogs) without source-specific rules.

3. Non-Destructive: Original intent is preserved through comments. For example:

   <!-- Original: <AnthropicCard> -->
   Card content here
   <!-- End: AnthropicCard -->
   

4. Automated: Zero user configuration required. The skill automatically detects and handles issues.

Warning Collection:

Collect all warnings during processing and include them in the final summary:

⚠️  Content Safety Processing:
  - Unknown component <AnthropicCard> (degraded to plain text)
  - Unknown component <FileGroup> (degraded to plain text)
  - Fixed <5k pattern (breaks MDX)
  - Injected 2 import statements
  - Mismatched <Callout> tags: 3 openings, 2 closings

📦 Injected Imports:
✅ import { Callout } from 'fumadocs-ui/components/callout';
✅ import { Cards, Card } from 'fumadocs-ui/components/card';

Benefits:

• ✅ Prevents 90% of MDX syntax errors from any source
• ✅ No manual cleanup needed for unknown components
• ✅ Clear visibility into what was changed and why
• ✅ Build never breaks due to imported content issues
• ✅ Diagnostics help identify patterns for future improvements

Step 3: Generate Article Slug

Create a URL-friendly slug from the article title:

• Convert to lowercase
• Replace spaces with hyphens
• Remove special characters
• Keep only: a-z, 0-9, hyphens
• Maximum 60 characters

Example: "Building React Apps with TypeScript" → "building-react-apps-with-typescript"

Step 3.5: Filter Content Images (HEURISTIC FILTERING)

Critical improvement: Most articles contain 15-20 images, but only 1-3 are actual content images (diagrams, charts, screenshots). The rest are decorative icons, logos, placeholders, or social preview images. We use heuristic rules to filter them.

Input: Array of image URLs (from Step 2)

Heuristic Filtering Rules:

// Blacklist - IMMEDIATE REJECTION
const blacklist = [
  'placeholder.svg',      // Placeholder images
  'favicon',              // Website icons
  'logo',                 // Company logos
  'spinner',              // Loading animations
  'avatar',               // User avatars
  'decoration',           // Decorative elements
  'icon-',                // Icon files
  'social-share',         // Social media preview
  'og-image',             // OpenGraph preview
  'twitter-card'          // Twitter card images
];

// Whitelist - MUST KEEP
const whitelist = [
  'diagram',              // Architecture diagrams
  'chart',                // Data visualizations
  'screenshot',           // UI screenshots
  'visualization',        // Data viz
  'architecture',         // System architecture
  'flowchart',            // Process flows
  'graph',                // Charts/graphs
  'timeline'              // Timeline graphics
];

function heuristicFilter(images: ImageInfo[]): ImageInfo[] {
  return images.filter(img => {
    const url = img.url.toLowerCase();
    const filename = img.filename.toLowerCase();

    // 🚫 BLACKLIST: Immediate rejection
    if (blacklist.some(term => url.includes(term) || filename.includes(term))) {
      return false; // Skip decorative images
    }

    // ✅ WHITELIST: Must keep
    if (whitelist.some(term => url.includes(term) || filename.includes(term))) {
      return true; // Keep content images
    }

    // 📏 FILE TYPE & SIZE RULES
    if (url.endsWith('.png') && img.fileSize > 10000) return true; // PNG > 10KB likely content
    if (url.endsWith('.jpg') && img.fileSize > 15000) return true; // JPG > 15KB likely content
    if (url.endsWith('.svg') && !url.includes('placeholder')) return true; // SVG (except placeholder)

    // 🎯 CONTEXT RULES
    // If image appears near keywords like "diagram", "figure", "example"
    if (isNearContext(img, ['diagram', 'figure', 'example', 'illustration'])) {
      return true;
    }

    return false; // Default: exclude if uncertain
  }).slice(0, 5); // MAX 5 images to avoid clutter
}

Example Filtering:

• Input: 18 images from claude.com blog
• Detected: 1 placeholder.svg (11 variations), 1 og-image.jpg, 5 decorative SVG icons, 1 MCP diagram PNG
• Filtered: Only 1 image kept (MCP diagram)
• Result: 94% reduction in noise

User Confirmation (shows transparency):

📊 Image Analysis Complete:
✅ Found 18 images total
🎯 Identified 1 content image (MCP protocol diagram)
🚫 Filtered 17 decorative/placeholder images

Image to download:
[Preview: https://cdn.../mcp-diagram.png]
Description: MCP protocol architecture diagram

Download? (Enter=yes, no=skip):

Step 3.6: AI-Powered Concept Extraction

Extract key technical concepts from article content using Claude AI. This enables intelligent cross-referencing and related article recommendations.

Why AI instead of rules:

• Rules can only match keywords (e.g., "Skills" as a word)
• AI understands context (e.g., "Skills" as a Claude feature vs. "skills" as general abilities)
• AI determines importance (main topic vs. mentioned in passing)
• AI extracts semantic meaning, not just patterns

AI Prompt:

const conceptExtractionPrompt = `
Read the following article and extract 5-10 key technical concepts.

Title: "${articleTitle}"
Content: """${articleContent.substring(0, 5000)}"""

For each concept, provide:
1. term: The exact term/concept name
2. definition: Brief explanation (1 sentence)
3. isMainTopic: true if this article primarily explains this concept, false if just mentions it
4. importance: Score 1-10 (how central this concept is to the article)

Output format:
\\
\\`\\`\\`json
{
  "concepts": [
    {
      "term": "Skills",
      "definition": "Claude's feature for saving and reusing instruction sets",
      "isMainTopic": true,
      "importance": 10
    }
  ]
}
\\`\\`\\`

**Example AI Decision**:
For the sentence "Claude's Skills feature helps you build agents":
- AI understands "Skills" is a proper noun (Claude feature)
- AI understands "agents" refers to AI agents
- AI judges importance based on context and article focus

**Execution**:
1. **Call Claude AI**:
   ```typescript
   const response = await askClaude(conceptExtractionPrompt);
   const { concepts } = JSON.parse(response);

2. Save concept extraction:

   mkdir -p "archive/concepts"
   

   writeJson(`archive/concepts/${articleSlug}.json`, {
     article: articleSlug,
     lang: languageCode,
     title: articleTitle,
     concepts: concepts
   });
   

3. Output:

   🤖 AI Concept Extraction Complete:
   ✅ Extracted ${concepts.length} concepts
   🎯 Main topics: ${concepts.filter(c => c.isMainTopic).map(c => c.term).join(', ')}
   📋 All concepts: ${concepts.map(c => `${c.term}(${c.importance})`).join(', ')}
   

AI Decision Examples:

Example 1:

Input: "Claude's Skills feature allows you to save prompts"
AI Output:
- term: "Skills"
- isMainTopic: true (文章主要讲解Skills)
- importance: 10

Example 2:

Input: "Using Python with Claude Code"
AI Output:
- term: "Python"
- isMainTopic: false (只是提到Python，不是专门讲Python)
- importance: 6

Example 3:

Input: "The Model Context Protocol (MCP) is a protocol"
AI Output:
- term: "MCP"
- definition: "Model Context Protocol, connects AI assistants to external systems"
- isMainTopic: true
- importance: 9

AI vs Rules Comparison:

Key AI Decision Points:

1. Term Recognition: AI understands proper nouns vs. common words
2. Context Understanding: AI reads surrounding text to understand meaning
3. Importance Scoring: AI weighs concepts based on article focus
4. Main Topic Detection: AI identifies what the article is primarily about

Step 3.7: AI-Powered Cross-Reference Insertion

Automatically insert links to related articles when concepts are mentioned. AI decides where and how many links to insert for natural reading flow.

Why AI instead of naive replacement:

• Naive: Replace first occurrence of "Skills" → may break reading flow
• AI: Understands paragraph structure, inserts where most helpful
• AI: Avoids over-linking (not every mention needs a link)
• AI: Skips titles, code blocks, and already-linked text

Prerequisites:

• Step 3.6 must be completed (concepts extracted)
• Concept index must exist (archive/concept-index.json)

AI Prompt:

const crossReferencePrompt = `
You are adding intelligent cross-references to a technical article.

Target article: "${articleTitle}"
Content: """${articleContent}"""

Relevant concepts from this article (from Step 3.6):
${JSON.stringify(concepts.filter(c => !c.isMainTopic), null, 2)}

For each concept, here is the authoritative article to link to:
${JSON.stringify(conceptIndex, null, 2)}

Task:
1. Identify where these concepts are FIRST mentioned in the content
2. Determine if linking would help the reader (skip if obvious or already explained)
3. Insert links naturally (don't break reading flow)
4. Limit to 3-5 links max (avoid over-linking)
5. NEVER link in: headings, code blocks, links, or quotes

Output format:
\\`\\`\\`json
{
  "enhancedContent": "content with links inserted",
  "linksInserted": [
    {
      "position": 125,
      "concept": "Skills",
      "targetArticle": "skills-explained",
      "context": "first mention in paragraph",
      "reasoning": "Reader may need background on Skills concept"
    }
  ]
}
\\`\\`\\`

**Example: Before and After AI Insertion**

**Before** (original content):
```markdown
## Building Agents with Skills

When you combine Skills with the Claude Agent SDK, you can create powerful workflows. Skills allow you to save and reuse instructions.

After AI Enhancement:

## Building Agents with Skills

When you combine [Skills](→skills-explained) with the Claude Agent SDK, you can create powerful workflows. Skills allow you to save and reuse instructions.

AI Decision:

• Linked "Skills" on first mention in the paragraph
• Didn't link "Agent" (already explained earlier in the article)
• Didn't link "SDK" (too generic, not a core concept)
• Only 1 link inserted (would be overwhelming to link everything)

Execution:

1. Call Claude AI:

   const response = await askClaude(crossReferencePrompt);
   const { enhancedContent, linksInserted } = JSON.parse(response);
   

2. Save link metadata:

   writeJson(`archive/links/${articleSlug}.json`, {
     article: articleSlug,
     lang: languageCode,
     totalLinks: linksInserted.length,
     links: linksInserted
   });
   

3. Output:

   🤖 AI Cross-Reference Insertion Complete:
   ✅ Analyzed ${concepts.length} concepts
   🎯 Inserted ${linksInserted.length} links
   📍 Positions: ${linksInserted.map(l => l.position).join(', ')}
   💡 AI reasoning: ${linksInserted.map(l => l.reasoning).join('; ')}
   

AI Decision Examples:

Example 1: Skip obvious concepts

Content: "The HTTP protocol is used for web requests"
AI Decision: Don't link "HTTP" (too generic, most developers know it)

Example 2: Link important concept on first mention

Content: "Claude's Skills feature allows you to..."
AI Decision: Link "Skills" on first mention (core concept, reader may need context)

Example 3: Don't over-link

Content: "Skills are powerful. Skills allow reuse. Skills improve consistency."
AI Decision: Only link first "Skills" (linking all three would be overwhelming)

Example 4: Skip in headings

Content: "## Skills Overview\n\nSkills are..."
AI Decision: Don't link "Skills" in the heading (breaks formatting)

Key AI Decision Points:

1. Context Understanding: AI reads surrounding text to determine if context is already clear
2. Reader Benefit: AI judges if linking would help understanding or be distracting
3. Position Selection: AI chooses first natural mention, not mechanical first occurrence
4. Link Density: AI limits total links to avoid overwhelming the reader
5. Natural Integration: AI ensures links flow naturally in the sentence

AI vs Naive Comparison:

Step 4: Process Images (Three Strategies)

Based on user's choice in Step 1 (image handling mode), use one of these strategies:

Strategy A: External URLs Only (No Download) - Recommended for CORS-Supporting Sites

When to use: Source website supports CORS (like Claude.com, GitHub, etc.)

Process:

// No download needed - just keep the original URLs
// MDX will reference external images directly

// Example output in MDX:
// Original: ![MCP Diagram](https://cdn.example.com/mcp-diagram.png)
// Final:    ![MCP Diagram](https://cdn.example.com/mcp-diagram.png) ← Unchanged!

#### Strategy B: Download to Local (Safe Option) - Use for Unknown/Complex Sites

**When to use**: Unknown source, no CORS support, or want offline availability

**Process**:

1. **User Confirmation** (show transparency):

📊 Image Strategy: Download to Local

✅ Found {total_images} total images on page 🎯 Identified {filtered_images} content images (diagrams/screenshots) 🚫 Filtered {skipped_images} decorative images (placeholders/icons)

Images to download:

1. [Preview URL: https://.../mcp-diagram.png] → Description: MCP protocol architecture diagram → Size: 142KB PNG

2. [Preview URL: https://.../data-flow.png] → Description: Data flow visualization → Size: 89KB PNG

Download these images? (Enter=yes, no=skip) [yes]:

2. **Create directory**:
```bash
mkdir -p "public/images/docs/{article-slug}"

3. Download each image (with retry logic):

   curl -f -L -o "public/images/docs/{slug}/{image-name}" "{image_url}" || \
   curl -f -L -o "public/images/docs/{slug}/{image-name}" "{image_url}" || \
   echo "⚠️  Failed to download: {image_url}"
   

4. Update MDX references:

   // Original
   ![MCP architecture](https://example.com/mcp-diagram.png)
   
   // Updated
   ![MCP architecture](/images/docs/{article-slug}/mcp-architecture.png)
   

5. Handle failures gracefully:

   • If download fails, keep original URL
   • Log failure
   • Report in summary

Pros: Works 100%, offline, full control Cons: Slower, uses storage, more complex

Strategy C: Auto-Detect (Best of Both Worlds)

When to use: You want the skill to automatically decide

Process:

1. Test first image for CORS support:

   curl -I "{first_image_url}" | grep -i "access-control"
   # If returns "access-control-allow-origin: *" → external mode
   # If returns nothing or error → download mode
   

2. Based on result, auto-switch:

   const hasCORS = checkCORS(firstImageUrl);
   
   if (hasCORS) {
     console.log("✅ Images support CORS → Using external URLs");
     useStrategyExternal();
   } else {
     console.log("❌ No CORS support → Downloading images locally");
     useStrategyDownload();
   }
   

3. Process all images with chosen strategy

Pros: Intelligent, optimal choice, hands-off Cons: Extra test step, might mis-detect edge cases

Example Test Result:

Testing: https://cdn.prod.website-files.com/68a44d.../619a5262.png

Response:
HTTP/2 200
access-control-allow-origin: *
access-control-allow-methods: GET, HEAD
access-control-allow-headers: *

→ ✅ CORS supported → Use external URLs

Decision Guide

Default: auto (intelligent detection)

Post-Import Verification:

• External strategy: Test article in browser, verify images load
• Download strategy: Check public/images/docs/{slug}/ for files
• Auto strategy: Check both (see which path was chosen)

Step 5: Process YouTube Videos

Detect and embed YouTube videos from article content:

Detection Patterns:

// Match YouTube URLs in content
const patterns = [
  // iframe embeds
  /<iframe[^>]*src="https?:\/\/(www\.)?youtube\.com\/embed\/([a-zA-Z0-9_-]+)"[^>]*>/g,
  // youtu.be short URLs
  /https?:\/\/youtu\.be\/([a-zA-Z0-9_-]+)/g,
  // youtube.com/watch URLs
  /https?:\/\/(www\.)?youtube\.com\/watch\?v=([a-zA-Z0-9_-]+)/g,
];

function extractYouTubeVideos(content: string): YouTubeVideo[] {
  const videos = [];
  for (const pattern of patterns) {
    const matches = content.matchAll(pattern);
    for (const match of matches) {
      const videoId = match[2] || match[3];
      videos.push({
        id: videoId,
        embedUrl: `https://www.youtube.com/embed/${videoId}`,
        watchUrl: `https://www.youtube.com/watch?v=${videoId}`,
        startTime: extractTimeParam(match[0]) // Handle &t=123s
      });
    }
  }
  return videos;
}

In MDX: Use Fumadocs Video Component

Option 1: Keep iframe (simplest, works everywhere):

<iframe
  width="100%"
  height="500"
  src="https://www.youtube.com/embed/VIDEO_ID"
  title="Video title"
  frameBorder="0"
  allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture"
  allowFullScreen>
</iframe>

Option 2: Use Fumadocs <Video> component (if available):

import { Video } from 'fumadocs-ui/components/video';

<Video
  src="https://www.youtube.com/watch?v=VIDEO_ID"
  title="Introduction to MCP"
/>

Auto-Processing Flow:

const videos = extractYouTubeVideos(content);

if (videos.length > 0) {
  console.log(`📺 Found ${videos.length} YouTube video(s)`);

  // Auto-embed: Replace YouTube URLs with iframes
  content = content.replace(youtubeRegex, (match) => {
    const videoId = extractVideoId(match);
    return generateEmbedCode(videoId);
  });
}

No thumbnail download needed (per user preference):

• iframe loads video from YouTube directly
• ✅ Saves local storage
• ✅ Always up-to-date
• ✅ Supports captions, quality selection, fullscreen
• ✅ Handles mobile responsive automatically

Example Output: In generated MDX, video section becomes:

## Video Introduction

<p className="video-wrapper">
  <iframe
    src="https://www.youtube.com/embed/dQw4w9WgXcQ"
    width="100%"
    height="500"
    title="MCP Protocol Overview"
  />
</p>

Continue reading...

Summary Report (auto-processed, no user interruption):

📺 Videos: 2 YouTube videos detected and embedded
   - Video 1: Introduction to MCP (6:23)
   - Video 2: Advanced MCP features (12:45)
   ✅ Embedded using iframe for compatibility

Step 6: Classify Article

Load references/classification-rules.md and analyze the article to determine:

1. Category (one of 8):

   • development
   • data
   • ai-ml
   • design
   • content
   • business
   • devops
   • security

2. Difficulty Level (one of 3):

   • beginner: Introductory content, basic concepts
   • intermediate: Requires some background knowledge
   • advanced: Complex, requires expertise

3. Tags (3-7 tags):

   • Extract technology stack (e.g., react, python, docker)
   • Identify tools and frameworks
   • Add relevant keywords
   • Use lowercase with hyphens (e.g., machine-learning)

Step 6.5: Generate Article Cover Illustration

Purpose: Automatically create a modern, theme-relevant SVG cover illustration for the article.

Why this matters:

• Visual appeal increases engagement and readability
• Consistent illustration style across all articles
• Saves time compared to manual design
• Automatically matches article theme and category

Process:

1. Invoke the philosophical-illustrator skill:

   • This skill generates modern, colorful SVG illustrations for technical content
   • Automatically selects color palette based on category
   • Creates theme-relevant visual metaphors

2. Prepare illustration context:

   Article Title: {translated_title}
   Category: {category}
   

…(truncated)