# Project Scraper

> Orchestrates project search across multiple sources

- Skill: `rotinom0/project-scraper` (Agent Skill)
- Install (CLI): `npx skillmds@latest add rotinom0/project-scraper`
- Raw SKILL.md: https://api.skillmd.com/api/skills/rotinom0/project-scraper/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: rotinoM0 (https://skillmd.com/u/rotinom0)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/rotinom0/project-scraper

---


# Project Scraper Skill

Orchestrates search across GitHub, Awesome lists, and curated sources. Logs every search for future reference.

## When to use
- `/discover` command
- Background project discovery

## Sources
- **GitHub Search API** — repositories by topic, language, stars, activity
- **Awesome lists** — curated lists on GitHub (awesome-react, awesome-nodejs, etc.)
- **Topic exploration** — discover related topics from high-scoring projects

## Deduplication
Projects may appear in multiple sources. Dedup by full-name (owner/repo).
Track seen projects in `project_scraper/seen_projects.json`.

## Search History

Every `/discover` run is logged to `project_scraper/search_history.json`. This file grows over time and becomes the agent's memory of past searches.

### What gets logged

Each search entry records:

```json
{
  "id": "2026-08-26T14:30:00",
  "query": {
    "topics": ["react", "state management"],
    "languages": ["TypeScript"],
    "min_stars": 100,
    "filters": {}
  },
  "context": {
    "skills_used": ["React", "TypeScript", "Node.js"],
    "interests_matched": ["web development", "developer tools"],
    "reason": "User asked for React state management libraries"
  },
  "results": {
    "total_found": 47,
    "top_picks": [
      {"repo": "zustandjs/zustand", "score": 92, "reason": "Perfect stack match, active community"},
      {"repo": "pmndrs/jotai", "score": 88, "reason": "Atomic model, good first issues"}
    ]
  },
  "user_feedback": {
    "liked": ["zustandjs/zustand"],
    "disliked": [],
    "notes": "Prefers minimal API, lightweight solutions"
  }
}
```

### How history influences future searches

1. **Avoid repetition**: skip repos already recommended (unless user asks to re-scan)
2. **Learn preferences**: if user consistently likes lightweight tools, boost similar projects
3. **Refine queries**: if "React" searches return too many boilerplates, add "production-ready" filter
4. **Track skill evolution**: if user starts searching for "Rust" after initially saying they don't know it, update inferred skills
5. **Identify patterns**: "user always rejects projects with >50k stars" → they prefer smaller, approachable projects

### History file structure

```
project_scraper/
├── search_history.json    # All past searches (gitignored)
├── seen_projects.json     # Dedup index (gitignored)
└── README.md              # This file (tracked)
```

### When to update history

- **Before searching**: load `search_history.json` to inform query strategy
- **After searching**: append the new search entry
- **After user feedback**: update the last entry's `user_feedback` field
- **On `/discover --fresh`**: still log the search, but ignore `seen_projects.json` for dedup

### History limits

Keep the last **50 search entries** in `search_history.json`. When the file exceeds 50 entries, archive older ones to `project_scraper/archive/` (also gitignored). This prevents the file from growing unbounded while preserving recent context.