# Hunt > hunt - Skill: `rohitguta2432/hunt` (Agent Skill, multi-file: 82 files) - Install (CLI): `npx skillmds@latest add rohitguta2432/hunt` - Raw SKILL.md: https://api.skillmd.com/api/skills/rohitguta2432/hunt/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Marketing & Growth - Author: rohitguta2432 (https://skillmd.com/u/rohitguta2432) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/rohitguta2432/hunt --- # hunt Daily lead hunting + auto-pitch pipeline for Rohit Raj (rohitraj.tech). ## MANDATORY FIRST STEP — load wiki context **Before any other action**, read `~/wiki/wiki/project-freelance-hunt.md`. Wiki is source-of-truth for: - Pipeline state changes + dated overhauls (e.g. 2026-05-16 Apify removal, 2026-05-18 japan-dev disable + 3 verify_picks patches) - Source efficiency rolling 7d metrics - Known issues + caveats - Daily metrics trend (5-day table) - Documented bad sources / domains / role-account patterns Use `Read` tool on `~/wiki/wiki/project-freelance-hunt.md` as first action. Cross-reference `~/wiki/raw/freelance-hunt-YYYY-MM-DD.md` for latest daily dump if relevant. Also check `~/.claude/projects/.../memory/freelance_hunt_*.md` for short-form per-session learnings. Why: skill code evolves frequently. Wiki captures decisions + reasons. Without loading wiki first, agent may reintroduce removed code (e.g. Apify) or miss new sub-commands. **Runtime auto-sync:** `auto.py` runs `apply_learnings.py` at startup (between `build_projects.py` and `auto_cut.py`). That script parses the wiki + memory files and idempotently augments `~/.freelance-hunt/disabled_sources.json` and `blocklist.json` with anything documented but not yet in runtime state. So decisions captured in wiki ("disable japan-dev") propagate automatically — no manual state-file edits needed. Skip with `--no-learn` if testing. ## Pitch policy (HARD RULES — set 2026-05-08, reinforced 2026-05-19) **Contract / project-based ONLY (set 2026-07-06).** Rohit has a full-time job — every pitch is for contract / project-based work, NEVER a full-time application, and must say so upfront so there's zero confusion on the recipient's side. Enforced in: subject lines (`send.py` — "contract" in match + adapt subjects), an explicit framing line in all body templates (match/adapt/sharp/cofounder), and the LLM-sharp system prompt (`enrich_pitch.py` FRAMING hard rule). Never remove these; never draft a pitch that reads like a job application. **No stack-gating.** Any recruiter hiring a software engineer = valid target. Pitch regardless of stack overlap with `projects.json`. Permissive on stack, strict on role type (SE only — `TECH_ROLE_KWS` + `REJECT_ROLE_KWS` filter in `scan.py:788`). **No project-dependency-gating.** Even if Rohit has never built anything in the job's domain (e.g., blockchain, embedded, gamedev), still pitch. The adapt-variant template is honest about this: it offers a paid trial in their stack rather than claiming false alignment. Never filter picks for "we haven't done that domain". **Always showcase current work.** Every pitch (match + adapt + sharp variants) MUST embed top-3 projects from `projects.json` + portfolio link `https://rohitraj.tech`. Templates use `{{PROJECT_LIST}}` token rendered by `send.render_project_list()`. Match variant excludes the matched project to avoid duplication. **`score` is a ranking signal, NOT a filter.** Higher score → match-variant (calls out a specific aligned project). Lower score → adapt-variant (polyglot, stack-agnostic). Both ship. **A score=0 pick is not a low-quality pick — it just needs the polyglot template instead of the project-aligned one.** Common framing trap: "100% of picks scored 0, this is broken" — no, it just means scoring inputs (projects.json) were under-populated. The picks themselves are still valid pitches. **Config locked:** - `auto.py` default `min_score=0` (DO NOT raise — gating reintroduces stack-bias) - Adapt variant carries polyglot pitch + project showcase - Match variant only fires when score ≥ 2 (genuine intersection); LLM-sharp variant fallback to templated render still includes PROJECT_LIST - SE-role filter (`is_relevant()` in scan.py) gates non-engineering JDs (sales/HR/janitor/etc) — keep - Do not add stack-coverage rules - Do not introduce "drop if no matching project" / "drop if score < N" filters ## Sub-commands ### auto (one-shot full pipeline — hunt + cofounder) Chains: hunt scan → enrich → prune → send, then cofounder scan → send. Logs to `~/.freelance-hunt/auto.log`. ``` python3 ~/.claude/skills/hunt/scripts/auto.py # full run (hunt only — cofounder OFF by default since 2026-07-06, its sources are all dead) python3 ~/.claude/skills/hunt/scripts/auto.py --dry-run # preview, no send python3 ~/.claude/skills/hunt/scripts/auto.py --with-cofounder # re-include cofounder phase (only once a working source exists) python3 ~/.claude/skills/hunt/scripts/auto.py --cofounder-only # cofounder only python3 ~/.claude/skills/hunt/scripts/auto.py --no-enrich # skip Apify enrichment python3 ~/.claude/skills/hunt/scripts/auto.py --skip-scan # use existing picks python3 ~/.claude/skills/hunt/scripts/auto.py --min-score=2 --enrich-max=80 ``` Auto-prunes junk emails (opt.out / unsubscribe / security / workday ATS / icims / greenhouse). ### enrich (manual trigger; auto runs in pipeline) Two-stage enrichment: 1. `enrich_firecrawl.py` — visit `apply_url`s in firecrawl_review.json + deep-crawl /about /contact /team /jobs /careers on same domain 2. `email_finder.py` — DIY email resolver (replaces Apify-based enrich, removed 2026-05-16) ``` python3 ~/.claude/skills/hunt/scripts/enrich_firecrawl.py [--max=50] [--workers=4] [--no-deep] python3 ~/.claude/skills/hunt/scripts/email_finder.py [--max=200] [--workers=4] [--no-verify] ``` `email_finder.py` (DIY, $0 — no Hunter/Snov/Apollo): - Reads `apify_review.json` + `firecrawl_review.json` items without email - Strategy 1: parse "I'm " / "regards, " from text → name permutations (first, first.last, firstlast, f.last, ...) - Strategy 2: derive company domain candidates from `company` field (acme.com / acme.io / acme.co / acme.ai / acme.tech / acme.app / acme.in) - Strategy 3: role-based emails — careers@, jobs@, hiring@, hr@, recruiting@, talent@, contact@ - MX-lookup each domain via dnspython (skip if no MX) - Catch-all detection: probe `znq--test@domain` first — if accepted = catch-all, can't verify - SMTP RCPT TO probe on non-catchall domains; cache verified+rejected at `email_finder_cache.json` - Skips public mail (gmail/outlook/yahoo) + job-board domains (lever.co/greenhouse.io/adzuna.com/etc) - Throttle 0.5s between probes; default 4 workers - `--no-verify` skips SMTP — picks best-guess (first.last@domain or hiring@domain). 44% domain hit-rate observed on Adzuna queue - Promotes resolved emails into `today_picks.json` after `is_relevant` filter ### adzuna (auto runs in scan.py) Free Adzuna API — 1000 calls/mo, 19 countries (gb/us/in/au/ca/de/fr/nl/sg/za/nz/pl/br/it/es/mx/ch/at/ru). Env: `ADZUNA_APP_ID` + `ADZUNA_APP_KEY` (signup at developer.adzuna.com). Default category `it-jobs`. Items with inline email → `raw_jobs`; rest → `apify_review.json` queue for email_finder. ``` python3 ~/.claude/skills/hunt/scripts/adzuna_fetch.py # standalone fetch ``` Env tuning: - `ADZUNA_COUNTRIES=us,gb,in,...` — comma list - `ADZUNA_QUERY=developer` — Adzuna `what` param (single keyword recommended; OR syntax broken) - `ADZUNA_CATEGORY=it-jobs` — Adzuna category slug (overrides text match) - `ADZUNA_MAX_PAGES=2` — pages per country (50 results/page) - `ADZUNA_MAX_DAYS_OLD=30` — date filter; 0 disables ### bigset (funded-startup leads — auto runs in run_hunt) Ingests a **Bigset** CSV export (funded startups + careers page) into the review queue, so funded-company leads flow through `email_finder` → `verify_picks` → `parallel_send` like any other source. [Bigset](https://github.com/tinyfish-io/bigset) turns a one-line prompt into a self-refreshing dataset; it runs **fully key-less on local Ollama + DuckDuckGo** (local fork documented in the wiki under "Bigset lead source"). Funded startups = fresh budget + actively hiring, often before a posting goes live. ``` python3 ~/.claude/skills/hunt/scripts/bigset_fetch.py [--csv=PATH] [--dry-run] [--max=N] ``` - **Input** (first that exists): `--csv=PATH` → `$BIGSET_CSV` → `~/.freelance-hunt/bigset/*.csv` → `~/.freelance-hunt/bigset_leads.csv`. No CSV → no-op exit 0 (like adzuna when keys unset). - **Source label** `bigset` (non-strict prefix, so `is_relevant` doesn't gate on a tech keyword; `text` is seeded with "hiring software engineers" for scoring anyway). - **Domain resolution** (the crux): careers URL on the company's own domain → used directly. **ATS hosts** (lever.co / greenhouse.io / ashbyhq.com / recruitee / bamboohr / workday / …) → company slug pulled from the URL → MX-checked `.{com,io,co,ai}`, then company-name guesses. The resolved domain becomes `apply_url` so `email_finder` resolves emails on the company's own domain — ATS domains are in `JOBBOARD_DOMAINS` and would otherwise be rejected. - **Idempotent**: `_append_review_queue` dedups on `apply_url`, so re-dropping the same CSV never double-queues. Generate the CSV: build a dataset in Bigset with a prompt like *"Startups that raised funding in the last 30 days, with what they do, funding stage, location, and careers page"*, export CSV, drop it at `~/.freelance-hunt/bigset_leads.csv`. **Honest limits**: Bigset rows carry no recruiter name → `email_finder` falls back to role-account emails (`careers@`/`hiring@`) that `verify_picks` prunes on non-Workspace domains. Strong **company** source, weak **person** source — yield is gated by those pre-existing pipeline constraints, not the fetcher. ### preflight (run first, before any send) Verify SPF/DKIM/DMARC on sending domain. Gmail spam-buckets unauthenticated bulk. ``` bash ~/.claude/skills/hunt/scripts/check_dns.sh rohitraj.tech ``` ### scan Fetch fresh jobs, score against project DB, output ranked top-10. ``` python3 ~/.claude/skills/hunt/scripts/scan.py ``` Output: `~/.freelance-hunt/today_picks.json` — 10 jobs with email + score + matched projects + pitch variant. ### send Approval gate → fire emails via Gmail SMTP. Cap 95/24h. Throttle 45-60s + jitter. ``` python3 ~/.claude/skills/hunt/scripts/send.py # serial render+send (~150min for 95 picks) python3 ~/.claude/skills/hunt/scripts/parallel_send.py # parallel render + serial send (Recommended, ~85min) ``` Reads `today_picks.json`, prints all drafts, waits for `y` confirm. Logs sends to `~/.freelance-hunt/sent.json`. ### verify-picks (Default in auto.py, added 2026-05-17) Pre-send email validator. Filters `today_picks.json` BEFORE `parallel_send.py` to prevent bounces. Pure DNS path — no API calls, no quota, ~4s for 100 picks. ``` python3 ~/.claude/skills/hunt/scripts/verify_picks.py [--workers=16] [--strict] ``` **Filters**: 1. Cross-check `blocklist.json` (bounces auto-added by `learn_replies.py`) 2. DNS MX lookup via dnspython — drop domains without mail server 3. Hardcoded disposable-domain list (60+ providers) 4. Heuristic: drop `role_account` (hiring@/jobs@/careers@/hr@/etc) on domains WITHOUT Google/Microsoft Workspace MX. Small companies without Workspace rarely create `hiring@` mailbox → top bounce source. 5. `--strict` also rejects role_account on Workspace MX (very conservative) ### company-dedup (Default in auto.py, added 2026-06-14) — company-level re-pitch gate Pre-send filter that runs **right after `verify_picks`, before `parallel_send`**. The email deduper (`scan.load_seen_emails` + `dedup_window.json`) dedupes by EXACT email, but `sent.json` has no company field — so `careers@acme.com` and `jason@acme.com` are two different leads and **the same company gets pitched twice**. This drops any pick whose **domain** was already pitched within the dedup window. ```bash python3 ~/.claude/skills/hunt/scripts/company_dedup.py stats # top companies by # pitches (≥2 = re-pitched) python3 ~/.claude/skills/hunt/scripts/company_dedup.py check # report re-pitches in today_picks.json (exit 1 if any) python3 ~/.claude/skills/hunt/scripts/company_dedup.py check --in-place # pipeline form: drop them from today_picks.json (.bak backup) ``` - **Excludes** free-mail (gmail/outlook/…) and multi-tenant ATS/newsletter domains (greenhouse/lever/substack/…) — one such domain ≠ one company, so collapsing them would wrongly suppress (esp. cofounder gmail leads). Tune the `EXCLUDE` set in the script if an aggregator (e.g. `arc.dev`, `japan-dev.com`) shows up as a false "re-pitch" in `stats`. - **Window** defaults to the active email dedup window (`dedup_window.json`, else 60d); override with `--window N`. - Non-fatal in `auto.py` — a filter error never blocks the send. **Cache**: `~/.freelance-hunt/verify_cache.json` — avoids re-resolving same email. Caches **facts only** (MX hosts + API verdict), never the keep/drop decision — that policy is recomputed every call (2026-06-06 lesson: a cached verdict silently defeats `--strict`). mailcheck.ai abandoned 2026-05-17 (IP rate-banned at 429 after first burst). DNS-only path = no quota, scales to thousands/run. **HTTPS email-verification (optional, added 2026-06-11) — the volume unlock.** Port 25 is blocked here, so `email_finder` can't SMTP-verify and emits best-guess `hiring@`/`recruiting@` addresses; the bounce-storm incidents forced `--strict` to blanket-drop them all (a typical run: 351 candidates → 2 sent). Set a verifier key and `verify_picks` will instead confirm each guessed mailbox over HTTPS and KEEP the deliverable ones: - Env (in `.env`, all optional; **no-op until `EMAIL_VERIFY_API_KEY` is set** — behaviour byte-identical to DNS-only without it): ``` EMAIL_VERIFY_PROVIDER=reoon # reoon | neverbounce | zerobounce EMAIL_VERIFY_API_KEY=... # free-tier key, no port 25 needed EMAIL_VERIFY_MAX=120 # per-run call cap (protects free quota; verdicts cached across runs) EMAIL_VERIFY_TIMEOUT=12 ``` - **Scope** — consulted ONLY for picks we'd otherwise drop as a *right-audience* role-account guess (`role_no_workspace`/`role_strict`, incl. `recruit*`/`talentacq*` via `ROLE_ISH_SUBSTRINGS`). `valid`→keep (`reason=api_verified`), `invalid`→drop (`api_invalid`), `unknown`/catch-all/error/over-cap→heuristic fallback. **Never** called for `junk`/`role_substring` wrong-audience drops (a deliverable `legal@`/`investors@` is still the wrong audience), `no_mx`, or already-kept picks — so credits buy only net-new deliverable volume. - **Fail-safe**: any API error/timeout/over-cap → `unknown` → the existing heuristic drop stands, so a flaky verifier can never *cause* a bounce. ### parallel-send (Default in auto.py, added 2026-05-16) LLM render bottleneck: serial `send.py` calls LLM per pick → 60min for 95 picks. **`parallel_send.py` uses `multiprocessing.Pool(4)`** to fork 4 worker processes that render in parallel — preview phase 60min → ~8min. SMTP send loop still serial (Gmail rate-limits parallel connections from same account). ``` python3 ~/.claude/skills/hunt/scripts/parallel_send.py [--workers=4] [--dry-run] [--no-yes] ``` **Pattern: always parallelize I/O-bound work.** Render = LLM API (I/O). MX lookup = DNS (I/O). SMTP probe = TCP (I/O). All forkable via `multiprocessing.Pool` or `ThreadPoolExecutor`. Codified for daily use — `auto.py` invokes `parallel_send.py` by default, never raw `send.py`. State: `~/.freelance-hunt/rendered_drafts.json` (pre-rendered drafts, resumable if SMTP fails). Log: `~/.freelance-hunt/parallel_send.log`. ### followup T+4d bump + T+10d breakup pitches against unanswered prior sends. Reads `sent.json`, finds entries at right age that haven't been replied/declined, fires same SMTP path. ``` python3 ~/.claude/skills/hunt/scripts/followup.py python3 ~/.claude/skills/hunt/scripts/followup.py --dry-run ``` Mark a thread as replied to stop follow-ups: edit `sent.json` and set `"replied": true` on the matching entry. Same for `"declined": true`. ### build-projects One-time scrape rohitraj.tech → `~/.freelance-hunt/projects.json`. Re-run monthly. ``` python3 ~/.claude/skills/hunt/scripts/build_projects.py ``` ### apply-learnings (auto runs at start of `auto.py`, added 2026-05-18) Parses `~/wiki/wiki/project-freelance-hunt.md` + `~/.claude/projects/.../memory/freelance_hunt_*.md` for documented bad sources / blocked emails / blocked domains. Idempotently adds to `disabled_sources.json` + `blocklist.json`. The wiki becomes truth-source: any decision captured there ("disable japan-dev", "block @forwarder-domain.com") propagates to runtime on next `auto.py` run — no manual state-file edits. Recognition rules: - Markdown table rows under a `## Known bad sources` (or similar) heading → first-column labels treated as disabled sources. - `\`source-label\`` in prose with disable-hints (`dead`, `disabled`, `permanently disabled`, `auto-cut`, etc.) within the same line/sentence → disabled. Anti-hints (`dependency`, `import`, `package`) suppress (so dnspython etc. don't get flagged). - Emails under an `## Email-level blocklist` (or similar) heading → unconditionally blocklisted. - `@domain.com` mentions near block-hints (`bounce`, `forwarder`, `fake`, `ada`) → added as domain-block entry. ``` python3 ~/.claude/skills/hunt/scripts/apply_learnings.py [--dry-run] ``` State: `~/.freelance-hunt/apply_learnings.log` (append-only history), `~/.freelance-hunt/learnings_applied.json` (last-applied digest for diff). Skip in auto.py: `python3 auto.py --no-learn`. ### status Show daily count, sent log, replies (manual mark for now). ``` cat ~/.freelance-hunt/sent.json | jq '.[-20:]' ``` ### learn-failures (auto runs at start of `auto.py`, after auto-cut) Parses `~/.freelance-hunt/auto.log` for repeated HTTP errors per source (404, 403/limit, timeouts). Any source failing ≥3 days within last 7d is appended to `disabled_sources.json`. Complements `auto-cut` which gates by email-yield, not transport health. `scan.py` honors disabled list at fetch-call time (early-return `[]`). ``` python3 ~/.claude/skills/hunt/scripts/learn_failures.py [--days=7] [--threshold=3] [--dry-run] ``` State: `~/.freelance-hunt/learn_failures.log` (append-only decisions). ### auto-tune-dedup (auto runs at start of `auto.py`) Auto-shrinks/expands the email dedup window based on yesterday's send count vs `MIN_DAILY_TARGET`. Bounds 14d–90d, step ±10d. Writes `~/.freelance-hunt/dedup_window.json` consumed by `scan.load_seen_emails()` and `enrich_apify.load_seen_emails()`. ``` python3 ~/.claude/skills/hunt/scripts/auto_tune_dedup.py [--target=N] [--dry-run] ``` State: `~/.freelance-hunt/auto_tune_dedup.log`. ### learn-replies (auto runs at start of `auto.py`) Connects to Gmail INBOX via IMAP (`GMAIL_USER` + `GMAIL_APP_PASSWORD`), scans last 30d for replies/bounces/auto-replies matching `sent.json` recipients. Marks each `sent.json` entry with `replied/reply_date/reply_subject/reply_snippet`, `bounced/bounce_reason`, or `autoreply` flags. Bounced emails auto-appended to `blocklist.json`. Per-source 30d reply+bounce rate written to `~/.freelance-hunt/source_reply_rate.json`. Filters: classifies by From (mailer-daemon/postmaster → bounce), Subject ("delivery status notification" → bounce), Subject/body ("out of office" / "automatic reply" → autoreply), else → genuine reply. ``` python3 ~/.claude/skills/hunt/scripts/learn_replies.py [--days=30] [--dry-run] ``` State: `~/.freelance-hunt/learn_replies.log`, `~/.freelance-hunt/source_reply_rate.json`. ### auto-cut (auto runs at start of `auto.py`) Reads `~/.freelance-hunt/source_health.json` rolling stats. Any source with `dead_streak ≥ 3d` AND `raw ≥ 5` gets appended to `~/.freelance-hunt/disabled_sources.json`. `scan.py` filters items from disabled sources at write time. Decisions logged to `~/.freelance-hunt/auto_cut.log`. ``` python3 ~/.claude/skills/hunt/scripts/auto_cut.py # apply cuts python3 ~/.claude/skills/hunt/scripts/auto_cut.py --dry-run # report only ``` ### discover-sources (auto runs at start of `auto.py`) Daily research for new sources. Multi-strategy discovery: - **4 curated awesome-list READMEs** (lukasz-madon/awesome-remote-job, tramcar/awesome-job-boards, remoteintech/remote-jobs, poteto/hiring-without-whiteboards) - **HN Algolia search** — queries like "best remote job board", "freelance platform 2026", "post job free" → community-recommended boards - **Reddit searches** in r/cscareerquestions, r/digitalnomad, r/forhire — "where to post" / "remote job site" threads - **Curated PROMOTE_SEED** — 10 known high-yield boards (hasjob, wellfound, cutshort, hirect, outerjoin, remoteleaf, echojobs, nodesk, europeremotely, remoteok.io) auto-promoted on first run, skip 3-trial ramp Extracts ~750-800 unique candidate domains. Tests 15 unseen domains/day for email yield. Promotes any candidate to `active` after **3 trial wins** (≥1 email per fetch). Promoted domains flow into scan via `generic_fetch.py` next run. ``` python3 ~/.claude/skills/hunt/scripts/discover_sources.py # full run python3 ~/.claude/skills/hunt/scripts/discover_sources.py --max=20 # cap tests python3 ~/.claude/skills/hunt/scripts/discover_sources.py --dry-run # log only ``` State: `~/.freelance-hunt/source_candidates.json` (per-domain trial/win counts + promoted flag). Daily report: `~/wiki/raw/source-discovery-YYYY-MM-DD.md`. ### top-up (auto runs after first send) Loops scan → enrich → send up to 5 times until `MIN_DAILY_TARGET` (default 80) hunt emails sent today. Each loop widens HN reach via `DEEP_SCAN_MONTHS` env (3 → 4 → 5 → 6 → 7 months back). Tolerates 1 zero-delta loop; bails after 2 consecutive zero-delta loops (supply truly exhausted) or target reached. Note: Gmail SMTP cold-send ceiling ~50-100/day before spam flagging — domain SPF/DKIM/DMARC must be clean. ``` python3 ~/.claude/skills/hunt/scripts/top_up.py # default target 80 python3 ~/.claude/skills/hunt/scripts/top_up.py --target=100 --max-loops=7 ``` State: `~/.freelance-hunt/top_up.log`. Env: `MIN_DAILY_TARGET=80` in `.env`. ### wiki-dump (auto runs at end of `auto.py`) Mines today's run + 7d rolling source health and writes a Karpathy-pattern raw source to `~/wiki/raw/freelance-hunt-YYYY-MM-DD.md`. Sections: - Pipeline summary (sent today, picks, queues) - **Top stacks / problems / roles / domains / seniority / locations / engagement / salaries** (regex token freq from JD text) - **Source efficiency — today** (raw / emails-found / sent per source) - **Source efficiency — 7d rolling** (send-rate %, dead-streak days, OK/ZERO/DEAD flag) - **Suggested cuts** — auto-flags sources with ≥3d zero-email streak + ≥5 raw items - **Auto-decisions today** — disabled sources, promoted sources from log - **Recruiter hiring patterns** — regex extraction of "looking for X", "must have Y", "experience with Z", "N+ years of …" phrases grouped by pattern - Score distribution histogram, top-5 picks with quotes, sent log, top-5 cofounder leads Final stdout prints **DAILY SUMMARY** + **MARKET ANALYTICS** box: top stack / domain / role / seniority / engagement / location + 3 salary samples + sent counts + dead source flags. Pipeline writes ONLY to `~/wiki/raw/`. Run **"compile my wiki"** to merge into `~/wiki/wiki/project-freelance-hunt.md` + rolling `trends-*.md` (LLM-driven per `~/wiki/CLAUDE.md`). State: `~/.freelance-hunt/source_health.json` keeps rolling 7d per-source stats. ``` python3 ~/.claude/skills/hunt/scripts/wiki_dump.py # write today's raw file python3 ~/.claude/skills/hunt/scripts/wiki_dump.py --date=2026-05-08 ``` ### wiki-compile (auto runs after wiki-dump) After `wiki_dump.py`, `auto.py` shells out to a headless Claude Code call to merge raw → compiled wiki: ``` claude --add-dir ~/wiki --permission-mode acceptEdits -p "compile my wiki" ``` Flag rationale: - `--add-dir ~/wiki` — `~/wiki/` lives outside auto.py cwd; without this the headless session denies tool access. - `--permission-mode acceptEdits` — non-interactive `-p` mode can't approve Write/Edit prompts; auto-accepts edits but still gates Bash. Safer than `--dangerously-skip-permissions`. Disable: `AUTO_COMPILE_WIKI=0` env. Override binary: `CLAUDE_BIN=/path/to/claude`. Hard timeout 10min — pipeline continues if compile times out or `claude` CLI not found. Known noise (cosmetic, doesn't fail rc): SessionEnd hooks from vercel + codex plugins log `SyntaxError: Unexpected token '?'` — older Node parser hits `??` operator on session shutdown. ## State files - `~/.freelance-hunt/projects.json` — project DB (name, stack, summary, url) - `~/.freelance-hunt/sent.json` — log of sent emails (recipient, subject, date, score) - `~/.freelance-hunt/today_picks.json` — current run's picks - `~/.freelance-hunt/blocklist.json` — emails to never contact - `~/.freelance-hunt/source_health.json` — rolling 7d per-source stats (raw, emails, sent, dead-streak) - `~/.freelance-hunt/disabled_sources.json` — auto-cut list (≥3d dead, ≥5 raw) + learn_failures cuts - `~/.freelance-hunt/source_failures.json` — per-source failure ledger (date list + last_error) - `~/.freelance-hunt/dedup_window.json` — auto-tuned dedup window (days, reason, updated) - `~/.freelance-hunt/demand_snapshot.json` — last-7d top stacks/domains, injected into pitch STACK_SNAPSHOT - `~/.freelance-hunt/source_reply_rate.json` — per-source 30d reply + bounce rate - `~/.freelance-hunt/source_candidates.json` — discovery trial state (domain → trials/wins/promoted) - `~/.freelance-hunt/auto_cut.log` — append-only auto-cut decisions - `~/.freelance-hunt/discover.log` — append-only discovery runs - `~/.freelance-hunt/email_finder_cache.json` — MX lookups + verified/rejected emails + catchall-domain flags (DIY finder, added 2026-05-16) - `~/.freelance-hunt/email_finder.log` — append-only emfind runs (processed/found/promoted counts) - `~/.freelance-hunt/apify_review.json` — review queue (legacy filename; now consumed by `email_finder.py`, written by scan.py + adzuna_fetch.py + bigset_fetch.py) - `~/.freelance-hunt/firecrawl_review.json` — Firecrawl review queue - `~/.freelance-hunt/bigset_leads.csv` (or `bigset/*.csv`) — Bigset CSV export drop-zone, read by `bigset_fetch.py` (input; you create it) - `~/.freelance-hunt/bigset_fetch.log` — append-only bigset ingest runs (parsed/queued/dropped counts) - `~/.freelance-hunt/.env` — `GMAIL_USER`, `GMAIL_APP_PASSWORD`, `ADZUNA_APP_ID`, `ADZUNA_APP_KEY`, `FIRECRAWL_API_KEY` - `~/wiki/raw/freelance-hunt-YYYY-MM-DD.md` — daily Karpathy-wiki source (auto-written) ## Config (.env) ``` GMAIL_USER=rohitgupta2432@gmail.com GMAIL_APP_PASSWORD= # DAILY_CAP and COFOUNDER_CAP removed 2026-05-06 — only Gmail SMTP rolling-24h hard cap (~95) still gates sends. ROLLING_24H_CAP=95 THROTTLE_MIN=45 THROTTLE_MAX=60 SENDER_NAME=Rohit Raj SITE_URL=https://rohitraj.tech GITHUB=rohitguta2432 TIMEZONE=flexible across timezones OVERLAP=5+ hrs overlap any TZ HOURS_PER_WEEK=30-40 # Apify integration — REMOVED 2026-05-16 (credits exhausted, no replenish). # Scripts archived as apify_fetch.py.removed-* and enrich_apify.py.removed-*. # Env vars commented as APIFY_*_REMOVED_20260516=... # Adzuna API — RESTORED 2026-05-16 (native fetch, no Apify wrapper, free 1000 calls/mo) ADZUNA_APP_ID=xxx # developer.adzuna.com free signup ADZUNA_APP_KEY=xxx ADZUNA_COUNTRIES=us,gb,in,au,ca,de,fr,nl,sg,za,nz,pl,br,it,es,mx,ch,at,ru ADZUNA_QUERY=developer # single keyword; OR syntax broken in API ADZUNA_CATEGORY=it-jobs # filters out medical/retail tech roles ADZUNA_MAX_PAGES=2 # 50/page = 100/country/run ADZUNA_MAX_DAYS_OLD=30 # 0 disables date filter # Firecrawl integration (optional) # Free tier: 500 credits/mo. Each board scrape ~5-15 credits. FIRECRAWL_API_KEY=fc-xxx FIRECRAWL_QUERY=python+developer+remote FIRECRAWL_LOCATION=Remote FIRECRAWL_MAX_CREDITS=100 FIRECRAWL_SOURCES=weworkremotely,wwr-backend,wwr-fullstack,arc # Daily target — top_up.py loops scan+enrich+send until reached MIN_DAILY_TARGET=100 # bumped 80→100 2026-05-16 (50 hunt + 50 cofounder target) # Scrapling anti-bot fallback (added 2026-05-31) — see "Scrapling fallback" section. # All optional; these are the defaults. Set USE_SCRAPLING=0 to hard-disable. USE_SCRAPLING=1 SCRAPLING_VENV=~/.freelance-hunt/.venv-scrapling SCRAPLING_TIMEOUT=90 # per-fetch subprocess timeout (s) SCRAPLING_MAX_LAUNCHES=60 # per-process browser-launch cap # SCRAPLING_PROXY=http://user:pass@host:port # only for IP-rate-limited sources ``` ## Pitch variants - **match** — used when ≥1 project from DB scores ≥2 keywords vs job. Lead with matching projects. - **adapt** — fallback when 0 match. Polyglot angle. Templates: `~/.claude/skills/hunt/templates/pitch_match.txt` + `pitch_adapt.txt`. ## Scrapling fallback (anti-bot bypass, added 2026-05-31) Hunt fetches via `urllib` first. When a source returns a hard block (HTTP 403/429/503/401/406) or a transport error, the fetch **auto-escalates to scrapling** — curl_cffi Chrome-TLS impersonation first, then headless Chromium (Cloudflare-solving) if that's still blocked. Pure fallback: never on the happy path, so it costs nothing when urllib succeeds, and the whole thing is a no-op unless the venv exists and `USE_SCRAPLING` is on. **Why:** Reddit hard-blocks its `.json` API for datacenter traffic — 403 even through a real browser — which had killed the entire cofounder phase ("0 sent", 2026-05-29). `old.reddit.com` HTML still renders via the stealth browser, so the Reddit fetchers fall back to parsing that. ### Install (one-time, isolated) `scrapling[all]` pulls Playwright + Chromium, so it lives in its OWN venv — deliberately NOT in the hunt runtime python (keeps the heavy browser stack off the scan hot path): ``` python3.12 -m venv ~/.freelance-hunt/.venv-scrapling ~/.freelance-hunt/.venv-scrapling/bin/pip install "scrapling[all]>=0.4.8" ~/.freelance-hunt/.venv-scrapling/bin/scrapling install --force # downloads Chromium ``` ### Architecture (3 new scripts) - `_scrapling_worker.py` — the ONLY file that imports scrapling. Runs under the venv interpreter; fetches one URL, writes the raw body to stdout. Modes: `get` (curl_cffi, raw bytes — JSON/RSS parse unchanged), `stealth` (browser + solve_cloudflare — walled HTML), `auto` (get → escalate on block/empty). - `stealth_fetch.py` — imported by hunt's runtime python (stdlib only). Shells out to the worker under the venv interpreter. `stealth_get(url, mode)` returns the body or `None` (never raises). Enforces a per-process browser-launch budget. - `reddit_fetch.py` — old.reddit listing parser + targeted post-body fetch. Cofounder leads need an email (which lives in post bodies, not listings), so it spends one body fetch per *title-matched* post — bounded by the launch budget. Wired into: `scan.py fetch_url()` (covers every RSS/JSON source that routes through it), `scan.py fetch_reddit_subs()`, and `cofounder.py reddit_search()` + `reddit_query_search()`. ### Env (all optional; defaults are sane — added to `.env` section below) ``` USE_SCRAPLING=1 # master switch (0 = hard-disable) SCRAPLING_VENV=~/.freelance-hunt/.venv-scrapling # venv dir SCRAPLING_TIMEOUT=90 # per-fetch subprocess timeout (s) SCRAPLING_MAX_LAUNCHES=60 # per-process browser-launch cap (bounds wall-clock) SCRAPLING_PROXY= # http://user:pass@host:port for IP-rate-limited sources ``` ### Honest limits (don't oversell this) - Restores **access + title-matching**, not magic. Reddit cofounder posts rarely expose a raw email (they say "DM me"), and the pipeline hard-requires an email — so lead yield stays gated by that pre-existing constraint. The net win is resilience (no more 403 death of the whole phase) + reachability of the posts that *do* carry an email, plus restored title-signal matching. - The old disabled feeds (`cryptojobslist`, `web3.career`, `aijobs`, `pangian`) are **404 — dead/moved URLs, not anti-bot walls** (re-verified 2026-05-31; the wiki's "Cloudflare 403" note is stale). Scrapling can't revive a 404, so they stay in `disabled_sources.json`. `fetch_url` only escalates on 403/429/503, not 404 — so no browser launch is wasted on them. - IP-based blocks (as opposed to UA/TLS-fingerprint) need a residential proxy via `SCRAPLING_PROXY`; the stealth browser alone won't beat a datacenter-IP ban. - Test the bridge manually: `python3 ~/.claude/skills/hunt/scripts/stealth_fetch.py stealth`. ## Sources ### Hunt (`scan.py`) **Parallel scan (2026-07-13):** all sources now fetch concurrently via a ThreadPoolExecutor(8) registry in `scan.py main()` (was sequential, 8-9 min → ≈ slowest single source). Registry tags skip auto-disabled sources at fetch time; Reddit has a circuit breaker (2 consecutive blocked subs → skip the rest that run). **Order** (Apify removed 2026-05-16): 1. HN/Reddit/Jobicy/Himalayas/RemoteOK — inline-email sources (highest yield) 2. WWR/Remotive/Arbeitnow/Builtin/4DayWeek/SkipTheDrive — RSS (low email yield, → review queue) 3. Generic (auto-promoted) — discover_sources.py promoted boards 4. **Adzuna API** (native fetch — 19 countries, free 1000 calls/mo) — items → review queue 5. Firecrawl — paid LLM scrape on WWR/arc.dev → review queue 6. **Bigset CSV** (funded-startup leads via `bigset_fetch.py`; runs in `auto.py run_hunt` after scan, not inside scan.py) — items → review queue **Active sources**: - HN "Who is Hiring" via Algolia API (current month + DEEP_SCAN_MONTHS deeper) - HN "Freelancer? Seeking freelancer?" via Algolia - hnhiring.com (URL format fixed 2026-05-16: `/-`) - RemoteOK API (`https://remoteok.com/api`) - Reddit subs: r/forhire, r/jobbit, r/remotejs, r/hireaprogrammer, r/freelance_forhire - WeWorkRemotely RSS (`/categories/remote-programming-jobs.rss`) - Remotive API (`/api/remote-jobs?category=software-dev`) - Jobicy API (`https://jobicy.com/api/v2/remote-jobs`) - Himalayas API (`https://himalayas.app/jobs/api`) - Working Nomads (`https://www.workingnomads.com/api/exposed_jobs/`) - Arbeitnow (`https://www.arbeitnow.com/api/job-board-api`) — EU/visa angle - Justjoin.it API (`https://api.justjoin.it/v2/user-panel/offers` — Version: 2 header; fixed 2026-05-16) - Landing.jobs, NoFluffJobs, BuiltIn - 4DayWeek.io RSS, SkipTheDrive RSS - **Adzuna API** (19 countries: gb/us/in/au/ca/de/fr/nl/sg/za/nz/pl/br/it/es/mx/ch/at/ru) - **japan-dev.com** (Japan, English-speaking tech jobs, ~hundreds, inline @japan-dev.com forwarder emails) - **wanted.co.kr** (Korea, JSON API, ~100/run, no inline email → review queue) - **Bigset CSV** (funded-startup leads — `bigset_fetch.py` ingests a Bigset export → review queue; ATS careers URLs resolved to the company's own mail domain; runs in `run_hunt`) - **Freelancer.com** (added 2026-07-13 — public no-key API `api/projects/0.1/projects/active/?query=`, 10 stack queries → `freelancer_com_projects.json` bid queue; in-platform contact, never emailed) - **YC jobs** (added 2026-07-13 — `ycombinator.com/jobs` server-rendered HTML → `yc_jobs.json` apply queue; founding-engineer roles ranked first; apply via Work at a Startup) **Action email (added 2026-07-13):** `action_email.py` mails the merged projects/founding-roles view (freelance_projects_view) to GMAIL_USER after each non-dry run — the review-gated leads are the highest-conversion output and were previously buried in state files. Disable: `ACTION_EMAIL=0`. **FREE-ONLY MODE (2026-07-13, user decision — do not add paid APIs):** reoon + Firecrawl keys disabled in .env (both accounts were at 0 credits; user chose free-only over recharging). Consequences: email channel = real found-on-page emails only (free MX heuristic); role-account guesses (`hiring@…`) drop by design — no verifier means they're bounce risks. The PRIMARY output is the free bid/apply queue (freelancer.com API + freelancermap + peopleperhour + YC founding roles) mailed daily by `action_email.py`. `verify_picks.py` still fails LOUD if a provider key is ever set but erroring. **Permanently disabled (URL dead/Cloudflare-walled, 2026-05-16)**: - cryptojobslist (Cloudflare 403) - aijobs.net (no feed exists) - web3.career (all paths 404) - pangian.com (404) - Pangian RSS — global remote - Justjoin.it API — PL/EU tech - Landing.jobs Atom feed — EU tech - NoFluffJobs JSON API — PL/EU tech - BuiltIn RSS — US city tech (dev-engineering + remote feeds) - **Firecrawl** (optional, env-gated via `FIRECRAWL_API_KEY`) — LLM-powered scrape, JSON-extract via prompt. **10x more efficient per credit than Apify** for job boards (tested 2026-05-05: Indeed returned 15/15 relevant Python jobs at 9 credits, vs Apify 100 items mostly janitorial). Free tier 500 credits/mo = ~55 board scrapes. Sources: indeed, wellfound, weworkremotely, remoteok, ycombinator. Items WITHOUT email → `~/.freelance-hunt/firecrawl_review.json`. Daily usage tracked at `~/.freelance-hunt/firecrawl_usage.json`. Capped via `FIRECRAWL_MAX_CREDITS`. - **Apify Actors** (optional, env-gated via `APIFY_TOKEN`) — Upwork, LinkedIn, Wellfound, Indeed via Actor IDs in env. Job boards strip emails → most Apify items go to manual-review queue at `~/.freelance-hunt/apify_review.json` (apply via URL). Items WITH email auto-flow into SMTP pipeline. Costs ~$0.0001–$0.015 per item. Cap via `APIFY_MAX_ITEMS` (some Actors ignore — verify per-Actor). Skips silently when token unset. #### Apify caveats - Indeed Actor (`valig/indeed-jobs-scraper`) returns generic feed; `position` filter often ignored. Use `APIFY_INDEED_COUNTRY=us|uk|in|...` for region. - LinkedIn / Wellfound Actors require subscription tier in some cases (403 on free tier). - Per-Actor input shapes differ — see `ACTOR_INPUT_ADAPTERS` in `apify_fetch.py`. Add adapters for new Actors there. - Manual review file: `cat ~/.freelance-hunt/apify_review.json | jq '.[].apply_url'` then apply manually. ### Cofounder (`cofounder.py`) - HN Algolia search — 16 query variants (looking-for-CTO / cofounder / founding engineer / technical partner / equity-for-engineer / etc.) - **Comments** (`tags=comment`) AND **stories+Show HN** (`tags=story`) both scanned — comment-only missed founders posting standalone "looking for CTO" stories. - Reddit cofounder subs (14): r/cofounder, r/startups, r/Entrepreneur, r/EntrepreneurRideAlong, r/SaaS, r/ycombinator, r/indiebiz, r/SideProject, r/IMadeThis, r/roastmystartup, r/smallbusiness, r/AlphaandBetausers, r/startup, r/indiehackers - Two passes per sub: (a) `/new.json` recent fetch, (b) per-sub search via `/search.json?q=&restrict_sr=on` for 6 query variants — finds older relevant posts /new misses. - Indie Hackers RSS (best-effort; IH rarely exposes raw emails) - BetaList feeds (best-effort; feed URLs return 404 since 2026-05; left wired in case they restore) - ProductHunt feeds (best-effort; newest launches) - **Stale re-engage**: `load_sent()` only blocks emails sent within last **45d** (`RECENT_SEND_WINDOW_DAYS`). Older sends fall out of the dedup window so cofounder pipeline can re-pitch with refreshed project list. Critical for survival — HN cofounder threads have ~30-active-poster ceiling at any time, full dedup exhausts in ~7d. - **Out-of-scope** (gated/JS-rendered, no public API): - YC Co-Founder Matching (`startupschool.org`) — needs login + Playwright - Wellfound (AngelList) cofounder listings — no email until apply-flow - X / Twitter "looking for technical cofounder" — API gated ## Efficiency review — 2026-07-06 (read before "why is volume low?") Funnel that day: 385 candidates → verify kept 19 (strict) → company_dedup → **4 sent** vs target 100. Root causes found + fixed: 1. **emfind dead-item loop (FIXED)** — `email_finder.py` pending filter didn't exclude terminal `exhausted`/`no_domain` statuses, so the first 500 queue slots were the same failed items re-tried every run (`e …(truncated)