Project Overview

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

🎯 Project Overview

Skill Seekers is the universal documentation preprocessor for AI systems. It transforms documentation websites, GitHub repositories, and PDFs into production-ready formats for 16+ platforms: RAG pipelines (LangChain, LlamaIndex, Haystack), vector databases (Pinecone, Chroma, Weaviate, FAISS, Qdrant), AI coding assistants (Cursor, Windsurf, Cline, Continue.dev), and LLM platforms (Claude, Gemini, OpenAI).

Current Version: v3.1.0-dev Python Version: 3.10+ required Status: Production-ready, published on PyPI Website: https://skillseekersweb.com/ - Browse configs, share, and access documentation

📚 Table of Contents

• First Time Here? - Start here!
• Quick Commands - Common workflows
• Architecture - How it works
• Development - Building & testing
• Testing - Test strategy
• Debugging - Troubleshooting
• Contributing - How to add features

👋 First Time Here?

Complete this 3-minute setup to start contributing:

# 1. Install package in editable mode (REQUIRED for development)
pip install -e .

# 2. Verify installation
python -c "import skill_seekers; print(skill_seekers.__version__)"  # Should print: 3.1.0-dev

# 3. Run a quick test
pytest tests/test_scraper_features.py::test_detect_language -v

# 4. You're ready! Pick a task from the roadmap:
# https://github.com/users/yusufkaraaslan/projects/2

Quick Navigation:

• Building/Testing → Development Commands
• Architecture → Core Design Pattern
• Common Issues → Common Pitfalls
• Contributing → See CONTRIBUTING.md

⚡ Quick Command Reference (Most Used)

First time setup:

pip install -e .  # REQUIRED before running tests or CLI

Running tests (NEVER skip - user requirement):

pytest tests/ -v  # All tests
pytest tests/test_scraper_features.py -v  # Single file
pytest tests/ --cov=src/skill_seekers --cov-report=html  # With coverage

Code quality checks (matches CI):

ruff check src/ tests/  # Lint
ruff format src/ tests/  # Format
mypy src/skill_seekers  # Type check

Common workflows:

# NEW unified create command (auto-detects source type)
skill-seekers create https://docs.react.dev/ -p quick
skill-seekers create facebook/react -p standard
skill-seekers create ./my-project -p comprehensive
skill-seekers create tutorial.pdf

# Legacy commands (still supported)
skill-seekers scrape --config configs/react.json
skill-seekers github --repo facebook/react
skill-seekers analyze --directory . --comprehensive

# Package for LLM platforms
skill-seekers package output/react/ --target claude
skill-seekers package output/react/ --target gemini

RAG Pipeline workflows:

# LangChain Documents
skill-seekers package output/react/ --format langchain

# LlamaIndex TextNodes
skill-seekers package output/react/ --format llama-index

# Haystack Documents
skill-seekers package output/react/ --format haystack

# ChromaDB direct upload
skill-seekers package output/react/ --format chroma --upload

# FAISS export
skill-seekers package output/react/ --format faiss

# Weaviate/Qdrant upload (requires API keys)
skill-seekers package output/react/ --format weaviate --upload
skill-seekers package output/react/ --format qdrant --upload

AI Coding Assistant workflows:

# Cursor IDE
skill-seekers package output/react/ --target claude
cp output/react-claude/SKILL.md .cursorrules

# Windsurf
cp output/react-claude/SKILL.md .windsurf/rules/react.md

# Cline (VS Code)
cp output/react-claude/SKILL.md .clinerules

# Continue.dev (universal IDE)
python examples/continue-dev-universal/context_server.py
# Configure in ~/.continue/config.json

Cloud Storage:

# Upload to S3
skill-seekers cloud upload --provider s3 --bucket my-skills output/react.zip

# Upload to GCS
skill-seekers cloud upload --provider gcs --bucket my-skills output/react.zip

# Upload to Azure
skill-seekers cloud upload --provider azure --container my-skills output/react.zip

🏗️ Architecture

Core Design Pattern: Platform Adaptors

The codebase uses the Strategy Pattern with a factory method to support 16 platforms across 4 categories:

src/skill_seekers/cli/adaptors/
├── __init__.py          # Factory: get_adaptor(target/format)
├── base.py              # Abstract base class
# LLM Platforms (3)
├── claude.py            # Claude AI (ZIP + YAML)
├── gemini.py            # Google Gemini (tar.gz)
├── openai.py            # OpenAI ChatGPT (ZIP + Vector Store)
# RAG Frameworks (3)
├── langchain.py         # LangChain Documents
├── llama_index.py       # LlamaIndex TextNodes
├── haystack.py          # Haystack Documents
# Vector Databases (5)
├── chroma.py            # ChromaDB
├── faiss_helpers.py     # FAISS
├── qdrant.py            # Qdrant
├── weaviate.py          # Weaviate
# AI Coding Assistants (4 - via Claude format + config files)
# - Cursor, Windsurf, Cline, Continue.dev
# Generic (1)
├── markdown.py          # Generic Markdown (ZIP)
└── streaming_adaptor.py # Streaming data ingest

Key Methods:

• package(skill_dir, output_path) - Platform-specific packaging
• upload(package_path, api_key) - Platform-specific upload (where applicable)
• enhance(skill_dir, mode) - AI enhancement with platform-specific models
• export(skill_dir, format) - Export to RAG/vector DB formats

Data Flow (5 Phases)

1. Scrape Phase (doc_scraper.py:scrape_all())

   • BFS traversal from base_url
   • Output: output/{name}_data/pages/*.json

2. Build Phase (doc_scraper.py:build_skill())

   • Load pages → Categorize → Extract patterns
   • Output: output/{name}/SKILL.md + references/*.md

3. Enhancement Phase (optional, enhance_skill_local.py)

   • LLM analyzes references → Rewrites SKILL.md
   • Platform-specific models (Sonnet 4, Gemini 2.0, GPT-4o)

4. Package Phase (package_skill.py → adaptor)

   • Platform adaptor packages in appropriate format
   • Output: .zip or .tar.gz

5. Upload Phase (optional, upload_skill.py → adaptor)

   • Upload via platform API

File Structure (src/ layout) - Key Files Only

src/skill_seekers/
├── cli/                              # All CLI commands
│   ├── main.py                       # ⭐ Git-style CLI dispatcher
│   ├── doc_scraper.py                # ⭐ Main scraper (~790 lines)
│   │   ├── scrape_all()              # BFS traversal engine
│   │   ├── smart_categorize()        # Category detection
│   │   └── build_skill()             # SKILL.md generation
│   ├── github_scraper.py             # GitHub repo analysis
│   ├── codebase_scraper.py           # ⭐ Local analysis (C2.x+C3.x)
│   ├── package_skill.py              # Platform packaging
│   ├── unified_scraper.py            # Multi-source scraping
│   ├── unified_codebase_analyzer.py  # Three-stream GitHub+local analyzer
│   ├── enhance_skill_local.py        # AI enhancement (LOCAL mode)
│   ├── enhance_status.py             # Enhancement status monitoring
│   ├── upload_skill.py               # Upload to platforms
│   ├── install_skill.py              # Complete workflow automation
│   ├── install_agent.py              # Install to AI agent directories
│   ├── pattern_recognizer.py         # C3.1 Design pattern detection
│   ├── test_example_extractor.py     # C3.2 Test example extraction
│   ├── how_to_guide_builder.py       # C3.3 How-to guide generation
│   ├── config_extractor.py           # C3.4 Configuration extraction
│   ├── generate_router.py            # C3.5 Router skill generation
│   ├── code_analyzer.py              # Multi-language code analysis
│   ├── api_reference_builder.py      # API documentation builder
│   ├── dependency_analyzer.py        # Dependency graph analysis
│   ├── signal_flow_analyzer.py       # C3.10 Signal flow analysis (Godot)
│   ├── pdf_scraper.py                # PDF extraction
│   └── adaptors/                     # ⭐ Platform adaptor pattern
│       ├── __init__.py               # Factory: get_adaptor()
│       ├── base_adaptor.py           # Abstract base
│       ├── claude_adaptor.py         # Claude AI
│       ├── gemini_adaptor.py         # Google Gemini
│       ├── openai_adaptor.py         # OpenAI ChatGPT
│       ├── markdown_adaptor.py       # Generic Markdown
│       ├── langchain.py              # LangChain RAG
│       ├── llama_index.py            # LlamaIndex RAG
│       ├── haystack.py               # Haystack RAG
│       ├── chroma.py                 # ChromaDB
│       ├── faiss_helpers.py          # FAISS
│       ├── qdrant.py                 # Qdrant
│       ├── weaviate.py               # Weaviate
│       └── streaming_adaptor.py      # Streaming data ingest
└── mcp/                              # MCP server (26 tools)
    ├── server_fastmcp.py             # FastMCP server
    └── tools/                        # Tool implementations

Most Modified Files (when contributing):

• Platform adaptors: src/skill_seekers/cli/adaptors/{platform}.py
• Tests: tests/test_{feature}.py
• Configs: configs/{framework}.json

🛠️ Development Commands

Setup

# Install in editable mode (required before tests due to src/ layout)
pip install -e .

# Install with all platform dependencies
pip install -e ".[all-llms]"

# Install specific platforms
pip install -e ".[gemini]"   # Google Gemini
pip install -e ".[openai]"   # OpenAI ChatGPT

Running Tests

CRITICAL: Never skip tests - User requires all tests to pass before commits.

# All tests (must run pip install -e . first!)
pytest tests/ -v

# Specific test file
pytest tests/test_scraper_features.py -v

# Multi-platform tests
pytest tests/test_install_multiplatform.py -v

# With coverage
pytest tests/ --cov=src/skill_seekers --cov-report=term --cov-report=html

# Single test
pytest tests/test_scraper_features.py::test_detect_language -v

# MCP server tests
pytest tests/test_mcp_fastmcp.py -v

Test Architecture:

• 46 test files covering all features
• CI Matrix: Ubuntu + macOS, Python 3.10-3.13
• 2,121 tests passing (current v3.1.0), up from 700+ in v2.x
• Must run pip install -e . before tests (src/ layout requirement)
• Tests include create command integration tests, CLI refactor E2E tests

Building & Publishing

# Build package (using uv - recommended)
uv build

# Or using build
python -m build

# Publish to PyPI
uv publish

# Or using twine
python -m twine upload dist/*

Testing CLI Commands

# Test configuration wizard (NEW: v2.7.0)
skill-seekers config --show                          # Show current configuration
skill-seekers config --github                        # GitHub token setup
skill-seekers config --test                          # Test connections

# Test resume functionality (NEW: v2.7.0)
skill-seekers resume --list                          # List resumable jobs
skill-seekers resume --clean                         # Clean up old jobs

# Test GitHub scraping with profiles (NEW: v2.7.0)
skill-seekers github --repo facebook/react --profile personal    # Use specific profile
skill-seekers github --repo owner/repo --non-interactive         # CI/CD mode

# Test scraping (dry run)
skill-seekers scrape --config configs/react.json --dry-run

# Test codebase analysis (C2.x features)
skill-seekers analyze --directory . --output output/codebase/

# Test pattern detection (C3.1)
skill-seekers patterns --file src/skill_seekers/cli/code_analyzer.py

# Test how-to guide generation (C3.3)
skill-seekers how-to-guides output/test_examples.json --output output/guides/

# Test enhancement status monitoring
skill-seekers enhance-status output/react/ --watch

# Test multi-platform packaging
skill-seekers package output/react/ --target gemini --dry-run

# Test MCP server (stdio mode)
python -m skill_seekers.mcp.server_fastmcp

# Test MCP server (HTTP mode)
python -m skill_seekers.mcp.server_fastmcp --transport http --port 8765

New v3.0.0 CLI Commands

# Setup wizard (interactive configuration)
skill-seekers-setup

# Cloud storage operations
skill-seekers cloud upload --provider s3 --bucket my-bucket output/react.zip
skill-seekers cloud download --provider gcs --bucket my-bucket react.zip
skill-seekers cloud list --provider azure --container my-container

# Embedding server (for RAG pipelines)
skill-seekers embed --port 8080 --model sentence-transformers

# Sync & incremental updates
skill-seekers sync --source https://docs.react.dev/ --target output/react/
skill-seekers update --skill output/react/ --check-changes

# Quality metrics & benchmarking
skill-seekers quality --skill output/react/ --report
skill-seekers benchmark --config configs/react.json --compare-versions

# Multilingual support
skill-seekers multilang --detect output/react/
skill-seekers multilang --translate output/react/ --target zh-CN

# Streaming data ingest
skill-seekers stream --source docs/ --target output/streaming/

🔧 Key Implementation Details

CLI Architecture (Git-style)

Entry point: src/skill_seekers/cli/main.py

The unified CLI modifies sys.argv and calls existing main() functions to maintain backward compatibility:

# Example: skill-seekers scrape --config react.json
# Transforms to: doc_scraper.main() with modified sys.argv

Subcommands: create, scrape, github, pdf, unified, codebase, enhance, enhance-status, package, upload, estimate, install, install-agent, patterns, how-to-guides

NEW: Unified create Command

The recommended way to create skills - Auto-detects source type and provides progressive help disclosure:

# Auto-detection examples
skill-seekers create https://docs.react.dev/         # → Web scraping
skill-seekers create facebook/react                  # → GitHub analysis
skill-seekers create ./my-project                    # → Local codebase
skill-seekers create tutorial.pdf                    # → PDF extraction
skill-seekers create configs/react.json              # → Multi-source

# Progressive help system
skill-seekers create --help           # Shows universal args only (13 flags)
skill-seekers create --help-web       # Shows web-specific options
skill-seekers create --help-github    # Shows GitHub-specific options
skill-seekers create --help-local     # Shows local analysis options
skill-seekers create --help-pdf       # Shows PDF extraction options
skill-seekers create --help-advanced  # Shows advanced/rare options
skill-seekers create --help-all       # Shows all 120+ flags

# Universal flags work for ALL sources
skill-seekers create <source> -p quick                    # Preset (-p shortcut)
skill-seekers create <source> --enhance-level 2           # AI enhancement (0-3)
skill-seekers create <source> --chunk-for-rag             # RAG chunking
skill-seekers create <source> --dry-run                   # Preview

Key improvements:

• Single command replaces scrape/github/analyze for most use cases
• Smart detection - No need to specify source type
• Progressive disclosure - Default help shows 13 flags, detailed help available
• -p shortcut - Quick preset selection (-p quick|standard|comprehensive)
• Universal features - RAG chunking, dry-run, presets work everywhere

Recent Additions:

• create - NEW: Unified command with auto-detection and progressive help
• codebase - Local codebase analysis without GitHub API (C2.x + C3.x features)
• enhance-status - Monitor background/daemon enhancement processes
• patterns - Detect design patterns in code (C3.1)
• how-to-guides - Generate educational guides from tests (C3.3)

Platform Adaptor Usage

from skill_seekers.cli.adaptors import get_adaptor

# Get platform-specific adaptor
adaptor = get_adaptor('gemini')  # or 'claude', 'openai', 'markdown'

# Package skill
adaptor.package(skill_dir='output/react/', output_path='output/')

# Upload to platform
adaptor.upload(
    package_path='output/react-gemini.tar.gz',
    api_key=os.getenv('GOOGLE_API_KEY')
)

# AI enhancement
adaptor.enhance(skill_dir='output/react/', mode='api')

C3.x Codebase Analysis Features

The project has comprehensive codebase analysis capabilities (C3.1-C3.8):

C3.1 Design Pattern Detection (pattern_recognizer.py):

• Detects 10 common patterns: Singleton, Factory, Observer, Strategy, Decorator, Builder, Adapter, Command, Template Method, Chain of Responsibility
• Supports 9 languages: Python, JavaScript, TypeScript, C++, C, C#, Go, Rust, Java
• Three detection levels: surface (fast), deep (balanced), full (thorough)
• 87% precision, 80% recall on real-world projects

C3.2 Test Example Extraction (test_example_extractor.py):

• Extracts real usage examples from test files
• Categories: instantiation, method_call, config, setup, workflow
• AST-based for Python, regex-based for 8 other languages
• Quality filtering with confidence scoring

C3.3 How-To Guide Generation (how_to_guide_builder.py):

• Transforms test workflows into educational guides
• 5 AI enhancements: step descriptions, troubleshooting, prerequisites, next steps, use cases
• Dual-mode AI: API (fast) or LOCAL (free with Claude Code Max)
• 4 grouping strategies: AI tutorial group, file path, test name, complexity

C3.4 Configuration Pattern Extraction (config_extractor.py):

• Extracts configuration patterns from codebases
• Identifies config files, env vars, CLI arguments
• AI enhancement for better organization

C3.5 Architectural Overview (generate_router.py):

• Generates comprehensive ARCHITECTURE.md files
• Router skill generation for large documentation
• Quality improvements: 6.5/10 → 8.5/10 (+31%)
• Integrates GitHub metadata, issues, labels

C3.6 AI Enhancement (Claude API integration):

• Enhances C3.1-C3.5 with AI-powered insights
• Pattern explanations and improvement suggestions
• Test example context and best practices
• Guide enhancement with troubleshooting and prerequisites

C3.7 Architectural Pattern Detection (architectural_pattern_detector.py):

• Detects 8 architectural patterns (MVC, MVVM, MVP, Repository, etc.)
• Framework detection (Django, Flask, Spring, React, Angular, etc.)
• Multi-file analysis with directory structure patterns
• Evidence-based detection with confidence scoring

C3.8 Standalone Codebase Scraper (codebase_scraper.py):

# Quick analysis (1-2 min, basic features only)
skill-seekers analyze --directory /path/to/repo --quick

# Comprehensive analysis (20-60 min, all features + AI)
skill-seekers analyze --directory . --comprehensive

# With AI enhancement (auto-detects API or LOCAL)
skill-seekers analyze --directory . --enhance

# Granular AI enhancement control (NEW)
skill-seekers analyze --directory . --enhance-level 1  # SKILL.md only
skill-seekers analyze --directory . --enhance-level 2  # + Architecture + Config + Docs
skill-seekers analyze --directory . --enhance-level 3  # Full enhancement (all features)

# Disable specific features
skill-seekers analyze --directory . --skip-patterns --skip-how-to-guides

• Generates 300+ line standalone SKILL.md files from codebases
• All C3.x features integrated (patterns, tests, guides, config, architecture, docs)
• Complete codebase analysis without documentation scraping
• NEW: Granular AI enhancement control with --enhance-level (0-3)

C3.9 Project Documentation Extraction (codebase_scraper.py):

• Extracts and categorizes all markdown files from the project
• Auto-detects categories: overview, architecture, guides, workflows, features, etc.
• Integrates documentation into SKILL.md with summaries
• AI enhancement (level 2+) adds topic extraction and cross-references
• Controlled by depth: surface=raw copy, deep=parse+summarize, full=AI-enhanced
• Default ON, use --skip-docs to disable

C3.10 Signal Flow Analysis for Godot Projects (signal_flow_analyzer.py):

• Complete signal flow analysis system for event-driven Godot architectures
• Signal declaration extraction (detects signal keyword declarations)
• Connection mapping (tracks .connect() calls with targets and methods)
• Emission tracking (finds .emit() and emit_signal() calls)
• Real-world metrics: 208 signals, 634 connections, 298 emissions in test project
• Signal density metrics (signals per file)
• Event chain detection (signals triggering other signals)
• Signal pattern detection:
  • EventBus Pattern (0.90 confidence): Centralized signal hub in autoload
  • Observer Pattern (0.85 confidence): Multi-observer signals (3+ listeners)
  • Event Chains (0.80 confidence): Cascading signal propagation
• Signal-based how-to guides (C3.10.1):
  • AI-generated step-by-step usage guides (Connect → Emit → Handle)
  • Real code examples from project
  • Common usage locations
  • Parameter documentation
• Outputs: signal_flow.json, signal_flow.mmd (Mermaid diagram), signal_reference.md, signal_how_to_guides.md
• Comprehensive Godot 4.x support:
  • GDScript (.gd), Scene files (.tscn), Resources (.tres), Shaders (.gdshader)
  • GDScript test extraction (GUT, gdUnit4, WAT frameworks)
  • 396 test cases extracted in test project
  • Framework detection (Unity, Unreal, Godot)

Key Architecture Decision (BREAKING in v2.5.2):

• Changed from opt-in (--build-*) to opt-out (--skip-*) flags
• All analysis features now ON by default for maximum value
• Backward compatibility warnings for deprecated flags

Smart Categorization Algorithm

Located in doc_scraper.py:smart_categorize():

• Scores pages against category keywords
• 3 points for URL match, 2 for title, 1 for content
• Threshold of 2+ for categorization
• Auto-infers categories from URL segments if none provided
• Falls back to "other" category

Language Detection

Located in doc_scraper.py:detect_language():

1. CSS class attributes (language-*, lang-*)
2. Heuristics (keywords like def, const, func)

Configuration File Structure

Configs (configs/*.json) define scraping behavior:

{
  "name": "framework-name",
  "description": "When to use this skill",
  "base_url": "https://docs.example.com/",
  "selectors": {
    "main_content": "article",  // CSS selector
    "title": "h1",
    "code_blocks": "pre code"
  },
  "url_patterns": {
    "include": ["/docs"],
    "exclude": ["/blog"]
  },
  "categories": {
    "getting_started": ["intro", "quickstart"],
    "api": ["api", "reference"]
  },
  "rate_limit": 0.5,
  "max_pages": 500
}

🧪 Testing Guidelines

Test Coverage Requirements

• Core features: 100% coverage required
• Platform adaptors: Each platform has dedicated tests
• MCP tools: All 18 tools must be tested
• Integration tests: End-to-end workflows

Test Markers (from pytest.ini_options)

The project uses pytest markers to categorize tests:

# Run only fast unit tests (default)
pytest tests/ -v

# Include slow tests (>5 seconds)
pytest tests/ -v -m slow

# Run integration tests (requires external services)
pytest tests/ -v -m integration

# Run end-to-end tests (resource-intensive, creates files)
pytest tests/ -v -m e2e

# Run tests requiring virtual environment setup
pytest tests/ -v -m venv

# Run bootstrap feature tests
pytest tests/ -v -m bootstrap

# Skip slow and integration tests (fastest)
pytest tests/ -v -m "not slow and not integration"

Test Execution Strategy

By default, only fast tests run. Use markers to control test execution:

# Default: Only fast tests (skip slow/integration/e2e)
pytest tests/ -v

# Include slow tests (>5 seconds)
pytest tests/ -v -m slow

# Include integration tests (requires external services)
pytest tests/ -v -m integration

# Include resource-intensive e2e tests (creates files)
pytest tests/ -v -m e2e

# Run ONLY fast tests (explicit)
pytest tests/ -v -m "not slow and not integration and not e2e"

# Run everything (CI does this)
pytest tests/ -v -m ""

When to use which:

• Local development: Default (fast tests only) - pytest tests/ -v
• Pre-commit: Fast tests - pytest tests/ -v
• Before PR: Include slow + integration - pytest tests/ -v -m "not e2e"
• CI validation: All tests run automatically

Key Test Files

• test_scraper_features.py - Core scraping functionality
• test_mcp_server.py - MCP integration (18 tools)
• test_mcp_fastmcp.py - FastMCP framework
• test_unified.py - Multi-source scraping
• test_github_scraper.py - GitHub analysis
• test_pdf_scraper.py - PDF extraction
• test_install_multiplatform.py - Multi-platform packaging
• test_integration.py - End-to-end workflows
• test_install_skill.py - One-command install
• test_install_agent.py - AI agent installation
• conftest.py - Test configuration (checks package installation)

🌐 Environment Variables

# Claude AI / Compatible APIs
# Option 1: Official Anthropic API (default)
export ANTHROPIC_API_KEY=sk-ant-...

# Option 2: GLM-4.7 Claude-compatible API (or any compatible endpoint)
export ANTHROPIC_API_KEY=your-api-key
export ANTHROPIC_BASE_URL=https://glm-4-7-endpoint.com/v1

# Google Gemini (optional)
export GOOGLE_API_KEY=AIza...

# OpenAI ChatGPT (optional)
export OPENAI_API_KEY=sk-...

# GitHub (for higher rate limits)
export GITHUB_TOKEN=ghp_...

# Private config repositories (optional)
export GITLAB_TOKEN=glpat-...
export GITEA_TOKEN=...
export BITBUCKET_TOKEN=...

All AI enhancement features respect these settings:

• enhance_skill.py - API mode SKILL.md enhancement
• ai_enhancer.py - C3.1/C3.2 pattern and test example enhancement
• guide_enhancer.py - C3.3 guide enhancement
• config_enhancer.py - C3.4 configuration enhancement
• adaptors/claude.py - Claude platform adaptor enhancement

Note: Setting ANTHROPIC_BASE_URL allows you to use any Claude-compatible API endpoint, such as GLM-4.7 (智谱 AI).

📦 Package Structure (pyproject.toml)

Entry Points

[project.scripts]
# Main unified CLI
skill-seekers = "skill_seekers.cli.main:main"

# Individual tool entry points (Core)
skill-seekers-config = "skill_seekers.cli.config_command:main"                # v2.7.0 Configuration wizard
skill-seekers-resume = "skill_seekers.cli.resume_command:main"                # v2.7.0 Resume interrupted jobs
skill-seekers-scrape = "skill_seekers.cli.doc_scraper:main"
skill-seekers-github = "skill_seekers.cli.github_scraper:main"
skill-seekers-pdf = "skill_seekers.cli.pdf_scraper:main"
skill-seekers-unified = "skill_seekers.cli.unified_scraper:main"
skill-seekers-codebase = "skill_seekers.cli.codebase_scraper:main"           # C2.x Local codebase analysis
skill-seekers-enhance = "skill_seekers.cli.enhance_skill_local:main"
skill-seekers-enhance-status = "skill_seekers.cli.enhance_status:main"       # Status monitoring
skill-seekers-package = "skill_seekers.cli.package_skill:main"
skill-seekers-upload = "skill_seekers.cli.upload_skill:main"
skill-seekers-estimate = "skill_seekers.cli.estimate_pages:main"
skill-seekers-install = "skill_seekers.cli.install_skill:main"
skill-seekers-install-agent = "skill_seekers.cli.install_agent:main"
skill-seekers-patterns = "skill_seekers.cli.pattern_recognizer:main"         # C3.1 Pattern detection
skill-seekers-how-to-guides = "skill_seekers.cli.how_to_guide_builder:main" # C3.3 Guide generation
skill-seekers-workflows = "skill_seekers.cli.workflows_command:main"         # NEW: Workflow preset management

# New v3.0.0 Entry Points
skill-seekers-setup = "skill_seekers.cli.setup_wizard:main"                  # NEW: v3.0.0 Setup wizard
skill-seekers-cloud = "skill_seekers.cli.cloud_storage_cli:main"             # NEW: v3.0.0 Cloud storage
skill-seekers-embed = "skill_seekers.embedding.server:main"                  # NEW: v3.0.0 Embedding server
skill-seekers-sync = "skill_seekers.cli.sync_cli:main"                       # NEW: v3.0.0 Sync & monitoring
skill-seekers-benchmark = "skill_seekers.cli.benchmark_cli:main"             # NEW: v3.0.0 Benchmarking
skill-seekers-stream = "skill_seekers.cli.streaming_ingest:main"             # NEW: v3.0.0 Streaming ingest
skill-seekers-update = "skill_seekers.cli.incremental_updater:main"          # NEW: v3.0.0 Incremental updates
skill-seekers-multilang = "skill_seekers.cli.multilang_support:main"         # NEW: v3.0.0 Multilingual
skill-seekers-quality = "skill_seekers.cli.quality_metrics:main"             # NEW: v3.0.0 Quality metrics

Optional Dependencies

Project uses PEP 735 [dependency-groups] (Python 3.13+):

• Replaces deprecated tool.uv.dev-dependencies
• Dev dependencies: [dependency-groups] dev = [...] in pyproject.toml
• Install with: pip install -e . (installs only core deps)
• Install dev deps: See CI workflow or manually install pytest, ruff, mypy

[project.optional-dependencies]
gemini = ["google-generativeai>=0.8.0"]
openai = ["openai>=1.0.0"]
all-llms = ["google-generativeai>=0.8.0", "openai>=1.0.0"]

[dependency-groups]  # PEP 735 (replaces tool.uv.dev-dependencies)
dev = [
    "pytest>=8.4.2",
    "pytest-asyncio>=0.24.0",
    "pytest-cov>=7.0.0",
    "coverage>=7.11.0",
]

🚨 Critical Development Notes

Must Run Before Tests

# REQUIRED: Install package before running tests
pip install -e .

# Why: src/ layout requires package installation
# Without this, imports will fail

Never Skip Tests

Per user instructions in ~/.claude/CLAUDE.md:

• "never skip any test. always make sure all test pass"
• All 2,121 tests must pass before commits (v3.1.0)
• Run full test suite: pytest tests/ -v
• New tests added for create command and CLI refactor work

Platform-Specific Dependencies

Platform dependencies are optional (install only what you need):

# Install specific platform support
pip install -e ".[gemini]"         # Google Gemini
pip install -e ".[openai]"         # OpenAI ChatGPT
pip install -e ".[chroma]"         # ChromaDB
pip install -e ".[weaviate]"       # Weaviate
pip install -e ".[s3]"             # AWS S3
pip install -e ".[gcs]"            # Google Cloud Storage
pip install -e ".[azure]"          # Azure Blob Storage
pip install -e ".[mcp]"            # MCP integration
pip install -e ".[all]"            # Everything (16 platforms + cloud + embedding)

# Or install from PyPI:
pip install skill-seekers[gemini]    # Google Gemini support
pip install skill-seekers[openai]    # OpenAI ChatGPT support
pip install skill-seekers[all-llms]  # All LLM platforms
pip install skill-seekers[chroma]    # ChromaDB support
pip install skill-seekers[weaviate]  # Weaviate support
pip install skill-seekers[s3]        # AWS S3 support
pip install skill-seekers[all]       # All optional dependencies

AI Enhancement Modes

AI enhancement transforms basic skills (2-3/10) into production-ready skills (8-9/10). Two modes available:

API Mode (default if ANTHROPIC_API_KEY is set):

• Direct Claude API calls (fast, efficient)
• Cost: ~$0.15-$0.30 per skill
• Perfect for CI/CD automation
• Requires: export ANTHROPIC_API_KEY=sk-ant-...

LOCAL Mode (fallback if no API key):

• Uses Claude Code CLI (your existing Max plan)
• Free! No API charges
• 4 execution modes:
  • Headless (default): Foreground, waits for completion
  • Background (--background): Returns immediately
  • Daemon (--daemon): Fully detached with nohup
  • Terminal (--interactive-enhancement): Opens new terminal (macOS)
• Status monitoring: skill-seekers enhance-status output/react/ --watch
• Timeout configuration: --timeout 300 (seconds)

Enhancement Flag Consolidation (Phase 1)

IMPORTANT CHANGE: Three enhancement flags have been unified into a single granular control:

Old flags (deprecated):

• --enhance - Enable AI enhancement
• --enhance-local - Use LOCAL mode (Claude Code)
• --api-key KEY - Anthropic API key

New unified flag:

• --enhance-level LEVEL - Granular AI enhancement control (0-3, default: 2)
  • 0 - Disabled, no AI enhancement
  • 1 - SKILL.md only (core documentation)
  • 2 - + Architecture + Config + Docs (default, balanced)
  • 3 - Full enhancement (all features, comprehensive)

Auto-detection: Mode (API vs LOCAL) is auto-detected:

• If ANTHROPIC_API_KEY is set → API mode
• Otherwise → LOCAL mode (Claude Code Max)

Examples:

# Auto-detect mode, default enhancement level (2)
skill-seekers create https://docs.react.dev/

# Disable enhancement
skill-seekers create facebook/react --enhance-level 0

# SKILL.md only (fast)
skill-seekers create ./my-project --enhance-level 1

# Full enhancement (comprehensive)
skill-seekers create tutorial.pdf --enhance-level 3

# Force LOCAL mode with specific level
skill-seekers enhance output/react/ --mode LOCAL --enhance-level 2

# Background with status monitoring
skill-seekers enhance output/react/ --background
skill-seekers enhance-status output/react/ --watch

Migration: Old flags still work with deprecation warnings, will be removed in v4.0.0.

See docs/ENHANCEMENT_MODES.md for detailed documentation.

Git Workflow

Git Workflow Notes:

• Main branch: main
• Development branch: development
• Always create feature branches from development
• Branch naming: feature/{task-id}-{description} or feature/{category}

To see current status: git status

CI/CD Pipeline

The project has GitHub Actions workflows in .github/workflows/:

tests.yml - Runs on every push and PR to main or development:

1. Lint Job (Python 3.12, Ubuntu):

   • ruff check src/ tests/ - Code linting with GitHub annotations
   • ruff format --check src/ tests/ - Format validation
   • mypy src/skill_seekers - Type checking (continue-on-error)

2. Test Job (Matrix):

   • OS: Ubuntu + macOS
   • Python: 3.10, 3.11, 3.12
   • Exclusions: macOS + Python 3.10 (speed optimization)
   • Steps:
     • Install dependencies + pip install -e .
     • Run CLI tests (scraper, config, integration)
     • Run MCP server tests
     • Generate coverage report → Upload to Codecov

3. Summary Job - Single status check for branch protection

   • Ensures both lint and test jobs succeed
   • Provides single "All Checks Complete" status

release.yml - Triggers on version tags (e.g., v2.9.0):

• Builds package with uv build
• Publishes to PyPI with uv publish
• Creates GitHub release

Local Pre-Commit Validation

Run the same checks as CI before pushing:

# 1. Code quality (matches lint job) - WITH AUTO-FIX
uvx ruff check --fix --unsafe-fixes src/ tests/  # Auto-fix issues
uvx ruff format src/ tests/                      # Auto-format
uvx ruff check src/ tests/                       # Verify clean
uvx ruff format --check src/ tests/              # Verify formatted
mypy src/skill_seekers

# 2. Tests (matches test job)
pip install -e .
pytest tests/ -v --cov=src/skill_seekers --cov-report=term

# 3. If all pass, you're good to push!
git add -A  # Stage any auto-fixes
git commit --amend --no-edit  # Add fixes to commit (or new commit)
git push origin feature/my-feature

Branch Protection Rules:

• main: Requires tests + 1 review, only maintainers merge
• development: Requires tests to pass, default target for PRs

Common CI Failure Patterns and Fixes

If CI fails after your changes, follow this debugging checklist:

# 1. Fix linting errors automatically
uvx ruff check --fix --unsafe-fixes src/ tests/

# 2. Fix formatting issues
uvx ruff format src/ tests/

# 3. Check for remaining issues
uvx ruff check src/ tests/
uvx ruff format --check src/ tests/

# 4. Verify tests pass locally
pip install -e .
pytest tests/ -v

# 5. Push fixes
git add -A
git commit -m "fix: resolve CI linting/formatting issues"
git push

Critical dependency patterns to check:

• MCP version mismatch: Ensure requirements.txt and pyproject.toml have matching MCP versions
• Missing module-level imports: If a tool file imports a module at top level (e.g., import yaml), that module MUST be in core dependencies
• Try/except ImportError: Silent failures in try/except blocks can hide missing dependencies

Timing-sensitive tests:

• Benchmark tests may fail on slower CI runners (macOS)
• If a test times out or exceeds threshold only in CI, consider relaxing the threshold
• Local passing doesn't guarantee CI passing for performance tests

🚨 Common Pitfalls & Solutions

1. Import Errors

Problem: ModuleNotFoundError: No module named 'skill_seekers'

Solution: Must install package first due to src/ layout

pip install -e .

Why: The src/ layout prevents imports from repo root. Package must be installed.

2. Tests Fail with "No module named..."

Problem: Package not installed in test environment

Solution: CI runs pip install -e . before tests - do the same locally

pip install -e .
pytest tests/ -v

3. Platform-Specific Dependencies Not Found

Problem: ModuleNotFoundError: No module named 'google.generativeai'

Solution: Install platform-specific dependencies

pip install -e ".[gemini]"   # For Gemini
pip install -e ".[openai]"   # For OpenAI
pip install -e ".[all-llms]" # For all platforms

4. Git Branch Confusion

Problem: PR targets main instead of development

Solution: Always create PRs targeting development branch

git checkout development
git pull upstream development
git checkout -b feature/my-feature
# ... make changes ...
git push origin feature/my-feature
# Create PR: feature/my-feature → development

Important: See CONTRIBUTING.md for complete branch workflow.

5. Tests Pass Locally But Fail in CI

Problem: Different Python version or missing dependency

Solution: Test with multiple Python versions locally

# CI tests: Python 3.10, 3.11, 3.12 on Ubuntu + macOS
# Use pyenv or docker to test locally:
pyenv install 3.10.13 3.11.7 3.12.1

pyenv local 3.10.13
pip install -e . && pytest tests/ -v

pyenv local 3.11.7
pip install -e . && pytest tests/ -v

pyenv local 3.12.1
pip install -e . && pytest tests/ -v

6. Enhancement Not Working

Problem: AI enhancement fails or hangs

Solutions:

# Check if API key is set
echo $ANTHROPIC_API_KEY

# Try LOCAL mode instead (uses Claude Code Max, no API key needed)
skill-seekers enhance output/react/ --mode LOCAL

# Monitor enhancement status for background jobs
skill-seekers enhance-status output/react/ --watch

7. Rate Limit Errors from GitHub

Problem: 403 Forbidden from GitHub API

Solutions:

# Check current rate limit
curl -H "Authorization: token $GITHUB_TOKEN" https://api.github.com/rate_limit

# Configure multiple GitHub profiles (recommended)
skill-seekers config --github

# Use specific profile
skill-seekers github --repo owner/repo --profile work

# Test all configured tokens
skill-seekers config --test

8. Confused About Command Options

Problem: "Too many flags!" or "Which flags work with which sources?"

Solution: Use the progressive disclosure help system in the create command:

# Start with universal options (13 flags)
skill-seekers create --help

# Need web scraping options?
skill-seekers create --help-web

# GitHub-specific flags?
skill-seekers create --help-github

# See ALL options (120+ flags)?
skill-seekers create --help-all

# Quick preset shortcut
skill-seekers create <source> -p quick
skill-seekers create <source> -p standard
skill-seekers create <source> -p comprehensive

Why: The create command shows only relevant flags by default to reduce cognitive load.

Legacy commands

…(truncated)