Project Scraper

Project Scraper Skill

Orchestrates search across GitHub, Awesome lists, and curated sources. Logs every search for future reference.

When to use

• /discover command
• Background project discovery

Sources

• GitHub Search API — repositories by topic, language, stars, activity
• Awesome lists — curated lists on GitHub (awesome-react, awesome-nodejs, etc.)
• Topic exploration — discover related topics from high-scoring projects

Deduplication

Projects may appear in multiple sources. Dedup by full-name (owner/repo). Track seen projects in project_scraper/seen_projects.json.

Search History

Every /discover run is logged to project_scraper/search_history.json. This file grows over time and becomes the agent's memory of past searches.

What gets logged

Each search entry records:

{
  "id": "2026-08-26T14:30:00",
  "query": {
    "topics": ["react", "state management"],
    "languages": ["TypeScript"],
    "min_stars": 100,
    "filters": {}
  },
  "context": {
    "skills_used": ["React", "TypeScript", "Node.js"],
    "interests_matched": ["web development", "developer tools"],
    "reason": "User asked for React state management libraries"
  },
  "results": {
    "total_found": 47,
    "top_picks": [
      {"repo": "zustandjs/zustand", "score": 92, "reason": "Perfect stack match, active community"},
      {"repo": "pmndrs/jotai", "score": 88, "reason": "Atomic model, good first issues"}
    ]
  },
  "user_feedback": {
    "liked": ["zustandjs/zustand"],
    "disliked": [],
    "notes": "Prefers minimal API, lightweight solutions"
  }
}

How history influences future searches

1. Avoid repetition: skip repos already recommended (unless user asks to re-scan)
2. Learn preferences: if user consistently likes lightweight tools, boost similar projects
3. Refine queries: if "React" searches return too many boilerplates, add "production-ready" filter
4. Track skill evolution: if user starts searching for "Rust" after initially saying they don't know it, update inferred skills
5. Identify patterns: "user always rejects projects with >50k stars" → they prefer smaller, approachable projects

History file structure

project_scraper/
├── search_history.json    # All past searches (gitignored)
├── seen_projects.json     # Dedup index (gitignored)
└── README.md              # This file (tracked)

When to update history

• Before searching: load search_history.json to inform query strategy
• After searching: append the new search entry
• After user feedback: update the last entry's user_feedback field
• On /discover --fresh: still log the search, but ignore seen_projects.json for dedup

History limits

Keep the last 50 search entries in search_history.json. When the file exceeds 50 entries, archive older ones to project_scraper/archive/ (also gitignored). This prevents the file from growing unbounded while preserving recent context.