# Dl Crawl

> 20 crawlers avec TokenBucket, CircuitBreaker, cursor persistence

- Skill: `thewatcher01/dl-crawl` (Agent Skill)
- Install (CLI): `npx skillmds@latest add thewatcher01/dl-crawl`
- Raw SKILL.md: https://api.skillmd.com/api/skills/thewatcher01/dl-crawl/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: TheWatcher01 (https://skillmd.com/u/thewatcher01)
- Updated: 2026-09-10
- Page: https://skillmd.com/skills/thewatcher01/dl-crawl

---


# Crawlers — Datalake Souverain

20 crawlers with shared patterns:
- **AsyncGenerator** for memory-efficient streaming
- **TokenBucket** rate limiting per source
- **withRetry** exponential backoff
- **CircuitBreaker** prevents cascading failures
- **Cursor persistence** resume on interrupt (dl_crawl_cursors)

## Commands
```bash
pnpm crawl -- --source sirene --department 31 --max 5 --dry-run
pnpm crawl -- --source all --persist
pnpm enrich -- --siren 123456789
pnpm enrich -- --batch --limit 10 --type association
pnpm process -- --cross-ref --score --limit 1000
```

## Key patterns
- SIREN = universal key across all sources
- Dry run: `--dry-run` flag on all CLIs
- Audit trail: dl_crawl_runs + dl_crawl_cursors
- Source config: src/config/sources.ts (20 sources + CircuitBreaker config)
- OSINT tools: src/config/tools.ts (12 tools)