Config Format Reference - Skill Seekers

Config Format Reference - Skill Seekers

Version: 3.1.0
Last Updated: 2026-02-16
Complete JSON configuration specification

Table of Contents

• Overview
• Single-Source Config
  • Documentation Source
  • GitHub Source
  • PDF Source
  • Local Source
• Unified (Multi-Source) Config
• Common Fields
• Selectors
• Categories
• URL Patterns
• Examples

Overview

Skill Seekers uses JSON configuration files to define scraping targets. There are two types:

Single-Source Config

Documentation Source

For scraping documentation websites.

{
  "name": "react",
  "base_url": "https://react.dev/",
  "description": "React - JavaScript library for building UIs",
  
  "start_urls": [
    "https://react.dev/learn",
    "https://react.dev/reference/react"
  ],
  
  "selectors": {
    "main_content": "article",
    "title": "h1",
    "code_blocks": "pre code"
  },
  
  "url_patterns": {
    "include": ["/learn/", "/reference/"],
    "exclude": ["/blog/", "/community/"]
  },
  
  "categories": {
    "getting_started": ["learn", "tutorial", "intro"],
    "api": ["reference", "api", "hooks"]
  },
  
  "rate_limit": 0.5,
  "max_pages": 300,
  "merge_mode": "claude-enhanced"
}

Documentation Fields

GitHub Source

For analyzing GitHub repositories.

{
  "name": "react-github",
  "type": "github",
  "repo": "facebook/react",
  "description": "React GitHub repository analysis",
  
  "enable_codebase_analysis": true,
  "code_analysis_depth": "deep",
  
  "fetch_issues": true,
  "max_issues": 100,
  "issue_labels": ["bug", "enhancement"],
  
  "fetch_releases": true,
  "max_releases": 20,
  
  "fetch_changelog": true,
  "analyze_commit_history": true,
  
  "file_patterns": ["*.js", "*.ts", "*.tsx"],
  "exclude_patterns": ["*.test.js", "node_modules/**"],
  
  "rate_limit": 1.0
}

GitHub Fields

PDF Source

For extracting content from PDF files.

{
  "name": "product-manual",
  "type": "pdf",
  "pdf_path": "docs/manual.pdf",
  "description": "Product documentation manual",
  
  "enable_ocr": false,
  "password": "",
  
  "extract_images": true,
  "image_output_dir": "output/images/",
  
  "extract_tables": true,
  "table_format": "markdown",
  
  "page_range": [1, 100],
  "split_by_chapters": true,
  
  "chunk_size": 1000,
  "chunk_overlap": 100
}

PDF Fields

Local Source

For analyzing local codebases.

{
  "name": "my-project",
  "type": "local",
  "directory": "./my-project",
  "description": "Local project analysis",
  
  "languages": ["Python", "JavaScript"],
  "file_patterns": ["*.py", "*.js"],
  "exclude_patterns": ["*.pyc", "node_modules/**", ".git/**"],
  
  "analysis_depth": "comprehensive",
  
  "extract_api": true,
  "extract_patterns": true,
  "extract_test_examples": true,
  "extract_how_to_guides": true,
  "extract_config_patterns": true,
  
  "include_comments": true,
  "include_docstrings": true,
  "include_readme": true
}

Local Fields

Unified (Multi-Source) Config

Combine multiple sources into one skill with conflict detection.

{
  "name": "react-complete",
  "description": "React docs + GitHub + examples",
  "merge_mode": "claude-enhanced",
  
  "sources": [
    {
      "type": "docs",
      "name": "react-docs",
      "base_url": "https://react.dev/",
      "max_pages": 200,
      "categories": {
        "getting_started": ["learn"],
        "api": ["reference"]
      }
    },
    {
      "type": "github",
      "name": "react-github",
      "repo": "facebook/react",
      "fetch_issues": true,
      "max_issues": 50
    },
    {
      "type": "pdf",
      "name": "react-cheatsheet",
      "pdf_path": "docs/react-cheatsheet.pdf"
    },
    {
      "type": "local",
      "name": "react-examples",
      "directory": "./react-examples"
    }
  ],
  
  "conflict_detection": {
    "enabled": true,
    "rules": [
      {
        "field": "api_signature",
        "action": "flag_mismatch"
      }
    ]
  },
  
  "output_structure": {
    "group_by_source": false,
    "cross_reference": true
  }
}

Unified Fields

Source Types in Unified Config

Each source in the sources array can be:

Common Fields

Fields available in all config types:

Selectors

CSS selectors for content extraction from HTML:

{
  "selectors": {
    "main_content": "article",
    "title": "h1",
    "code_blocks": "pre code",
    "navigation": "nav.sidebar",
    "breadcrumbs": "nav[aria-label='breadcrumb']",
    "next_page": "a[rel='next']",
    "prev_page": "a[rel='prev']"
  }
}

Default Selectors

If not specified, these defaults are used:

Categories

Map URL patterns to content categories:

{
  "categories": {
    "getting_started": [
      "intro", "tutorial", "quickstart", 
      "installation", "getting-started"
    ],
    "core_concepts": [
      "concept", "fundamental", "architecture",
      "principle", "overview"
    ],
    "api_reference": [
      "reference", "api", "method", "function",
      "class", "interface", "type"
    ],
    "guides": [
      "guide", "how-to", "example", "recipe",
      "pattern", "best-practice"
    ],
    "advanced": [
      "advanced", "expert", "performance",
      "optimization", "internals"
    ]
  }
}

Categories appear as sections in the generated SKILL.md.

URL Patterns

Control which URLs are included or excluded:

{
  "url_patterns": {
    "include": [
      "/docs/",
      "/guide/",
      "/api/",
      "/reference/"
    ],
    "exclude": [
      "/blog/",
      "/news/",
      "/community/",
      "/search",
      "?print=1",
      "/_static/",
      "/_images/"
    ]
  }
}

Pattern Rules

• Patterns are matched against the URL path
• Use * for wildcards: /api/v*/
• Use ** for recursive: /docs/**/*.html
• Exclude takes precedence over include

Examples

React Documentation

{
  "name": "react",
  "base_url": "https://react.dev/",
  "description": "React - JavaScript library for building UIs",
  "start_urls": [
    "https://react.dev/learn",
    "https://react.dev/reference/react",
    "https://react.dev/reference/react-dom"
  ],
  "selectors": {
    "main_content": "article",
    "title": "h1",
    "code_blocks": "pre code"
  },
  "url_patterns": {
    "include": ["/learn/", "/reference/", "/blog/"],
    "exclude": ["/community/", "/search"]
  },
  "categories": {
    "getting_started": ["learn", "tutorial"],
    "api": ["reference", "api"],
    "blog": ["blog"]
  },
  "rate_limit": 0.5,
  "max_pages": 300
}

Django GitHub

{
  "name": "django-github",
  "type": "github",
  "repo": "django/django",
  "description": "Django web framework source code",
  "enable_codebase_analysis": true,
  "code_analysis_depth": "deep",
  "fetch_issues": true,
  "max_issues": 100,
  "fetch_releases": true,
  "file_patterns": ["*.py"],
  "exclude_patterns": ["tests/**", "docs/**"]
}

Unified Multi-Source

{
  "name": "godot-complete",
  "description": "Godot Engine - docs, source, and manual",
  "merge_mode": "claude-enhanced",
  "sources": [
    {
      "type": "docs",
      "name": "godot-docs",
      "base_url": "https://docs.godotengine.org/en/stable/",
      "max_pages": 500
    },
    {
      "type": "github",
      "name": "godot-source",
      "repo": "godotengine/godot",
      "fetch_issues": false
    },
    {
      "type": "pdf",
      "name": "godot-manual",
      "pdf_path": "docs/godot-manual.pdf"
    }
  ]
}

Local Project

{
  "name": "my-api",
  "type": "local",
  "directory": "./my-api-project",
  "description": "My REST API implementation",
  "languages": ["Python"],
  "file_patterns": ["*.py"],
  "exclude_patterns": ["tests/**", "migrations/**"],
  "analysis_depth": "comprehensive",
  "extract_api": true,
  "extract_test_examples": true
}

Validation

Validate your config before scraping:

# Using CLI
skill-seekers scrape --config my-config.json --dry-run

# Using MCP tool
validate_config({"config": "my-config.json"})

See Also

• CLI Reference - Command reference
• Environment Variables - Configuration environment

For more examples, see configs/ directory in the repository