Hunt

hunt

Daily lead hunting + auto-pitch pipeline for Rohit Raj (rohitraj.tech).

MANDATORY FIRST STEP — load wiki context

Before any other action, read ~/wiki/wiki/project-freelance-hunt.md. Wiki is source-of-truth for:

• Pipeline state changes + dated overhauls (e.g. 2026-05-16 Apify removal, 2026-05-18 japan-dev disable + 3 verify_picks patches)
• Source efficiency rolling 7d metrics
• Known issues + caveats
• Daily metrics trend (5-day table)
• Documented bad sources / domains / role-account patterns

Use Read tool on ~/wiki/wiki/project-freelance-hunt.md as first action. Cross-reference ~/wiki/raw/freelance-hunt-YYYY-MM-DD.md for latest daily dump if relevant. Also check ~/.claude/projects/.../memory/freelance_hunt_*.md for short-form per-session learnings.

Why: skill code evolves frequently. Wiki captures decisions + reasons. Without loading wiki first, agent may reintroduce removed code (e.g. Apify) or miss new sub-commands.

Runtime auto-sync: auto.py runs apply_learnings.py at startup (between build_projects.py and auto_cut.py). That script parses the wiki + memory files and idempotently augments ~/.freelance-hunt/disabled_sources.json and blocklist.json with anything documented but not yet in runtime state. So decisions captured in wiki ("disable japan-dev") propagate automatically — no manual state-file edits needed. Skip with --no-learn if testing.

Pitch policy (HARD RULES — set 2026-05-08, reinforced 2026-05-19)

Contract / project-based ONLY (set 2026-07-06). Rohit has a full-time job — every pitch is for contract / project-based work, NEVER a full-time application, and must say so upfront so there's zero confusion on the recipient's side. Enforced in: subject lines (send.py — "contract" in match + adapt subjects), an explicit framing line in all body templates (match/adapt/sharp/cofounder), and the LLM-sharp system prompt (enrich_pitch.py FRAMING hard rule). Never remove these; never draft a pitch that reads like a job application.

No stack-gating. Any recruiter hiring a software engineer = valid target. Pitch regardless of stack overlap with projects.json. Permissive on stack, strict on role type (SE only — TECH_ROLE_KWS + REJECT_ROLE_KWS filter in scan.py:788).

No project-dependency-gating. Even if Rohit has never built anything in the job's domain (e.g., blockchain, embedded, gamedev), still pitch. The adapt-variant template is honest about this: it offers a paid trial in their stack rather than claiming false alignment. Never filter picks for "we haven't done that domain".

Always showcase current work. Every pitch (match + adapt + sharp variants) MUST embed top-3 projects from projects.json + portfolio link https://rohitraj.tech. Templates use {{PROJECT_LIST}} token rendered by send.render_project_list(). Match variant excludes the matched project to avoid duplication.

score is a ranking signal, NOT a filter. Higher score → match-variant (calls out a specific aligned project). Lower score → adapt-variant (polyglot, stack-agnostic). Both ship. A score=0 pick is not a low-quality pick — it just needs the polyglot template instead of the project-aligned one. Common framing trap: "100% of picks scored 0, this is broken" — no, it just means scoring inputs (projects.json) were under-populated. The picks themselves are still valid pitches.

Config locked:

• auto.py default min_score=0 (DO NOT raise — gating reintroduces stack-bias)
• Adapt variant carries polyglot pitch + project showcase
• Match variant only fires when score ≥ 2 (genuine intersection); LLM-sharp variant fallback to templated render still includes PROJECT_LIST
• SE-role filter (is_relevant() in scan.py) gates non-engineering JDs (sales/HR/janitor/etc) — keep
• Do not add stack-coverage rules
• Do not introduce "drop if no matching project" / "drop if score < N" filters

Sub-commands

auto (one-shot full pipeline — hunt + cofounder)

Chains: hunt scan → enrich → prune → send, then cofounder scan → send. Logs to ~/.freelance-hunt/auto.log.

python3 ~/.claude/skills/hunt/scripts/auto.py                  # full run (hunt only — cofounder OFF by default since 2026-07-06, its sources are all dead)
python3 ~/.claude/skills/hunt/scripts/auto.py --dry-run        # preview, no send
python3 ~/.claude/skills/hunt/scripts/auto.py --with-cofounder # re-include cofounder phase (only once a working source exists)
python3 ~/.claude/skills/hunt/scripts/auto.py --cofounder-only # cofounder only
python3 ~/.claude/skills/hunt/scripts/auto.py --no-enrich      # skip Apify enrichment
python3 ~/.claude/skills/hunt/scripts/auto.py --skip-scan      # use existing picks
python3 ~/.claude/skills/hunt/scripts/auto.py --min-score=2 --enrich-max=80

Auto-prunes junk emails (opt.out / unsubscribe / security / workday ATS / icims / greenhouse).

enrich (manual trigger; auto runs in pipeline)

Two-stage enrichment:

1. enrich_firecrawl.py — visit apply_urls in firecrawl_review.json + deep-crawl /about /contact /team /jobs /careers on same domain
2. email_finder.py — DIY email resolver (replaces Apify-based enrich, removed 2026-05-16)

python3 ~/.claude/skills/hunt/scripts/enrich_firecrawl.py [--max=50] [--workers=4] [--no-deep]
python3 ~/.claude/skills/hunt/scripts/email_finder.py    [--max=200] [--workers=4] [--no-verify]

email_finder.py (DIY, $0 — no Hunter/Snov/Apollo):

• Reads apify_review.json + firecrawl_review.json items without email
• Strategy 1: parse "I'm " / "regards, " from text → name permutations (first, first.last, firstlast, f.last, ...)
• Strategy 2: derive company domain candidates from company field (acme.com / acme.io / acme.co / acme.ai / acme.tech / acme.app / acme.in)
• Strategy 3: role-based emails — careers@, jobs@, hiring@, hr@, recruiting@, talent@, contact@
• MX-lookup each domain via dnspython (skip if no MX)
• Catch-all detection: probe znq-<ts>-test@domain first — if accepted = catch-all, can't verify
• SMTP RCPT TO probe on non-catchall domains; cache verified+rejected at email_finder_cache.json
• Skips public mail (gmail/outlook/yahoo) + job-board domains (lever.co/greenhouse.io/adzuna.com/etc)
• Throttle 0.5s between probes; default 4 workers
• --no-verify skips SMTP — picks best-guess (first.last@domain or hiring@domain). 44% domain hit-rate observed on Adzuna queue
• Promotes resolved emails into today_picks.json after is_relevant filter

adzuna (auto runs in scan.py)

Free Adzuna API — 1000 calls/mo, 19 countries (gb/us/in/au/ca/de/fr/nl/sg/za/nz/pl/br/it/es/mx/ch/at/ru). Env: ADZUNA_APP_ID + ADZUNA_APP_KEY (signup at developer.adzuna.com). Default category it-jobs. Items with inline email → raw_jobs; rest → apify_review.json queue for email_finder.

python3 ~/.claude/skills/hunt/scripts/adzuna_fetch.py   # standalone fetch

Env tuning:

• ADZUNA_COUNTRIES=us,gb,in,... — comma list
• ADZUNA_QUERY=developer — Adzuna what param (single keyword recommended; OR syntax broken)
• ADZUNA_CATEGORY=it-jobs — Adzuna category slug (overrides text match)
• ADZUNA_MAX_PAGES=2 — pages per country (50 results/page)
• ADZUNA_MAX_DAYS_OLD=30 — date filter; 0 disables

bigset (funded-startup leads — auto runs in run_hunt)

Ingests a Bigset CSV export (funded startups + careers page) into the review queue, so funded-company leads flow through email_finder → verify_picks → parallel_send like any other source. Bigset turns a one-line prompt into a self-refreshing dataset; it runs fully key-less on local Ollama + DuckDuckGo (local fork documented in the wiki under "Bigset lead source"). Funded startups = fresh budget + actively hiring, often before a posting goes live.

python3 ~/.claude/skills/hunt/scripts/bigset_fetch.py [--csv=PATH] [--dry-run] [--max=N]

• Input (first that exists): --csv=PATH → $BIGSET_CSV → ~/.freelance-hunt/bigset/*.csv → ~/.freelance-hunt/bigset_leads.csv. No CSV → no-op exit 0 (like adzuna when keys unset).
• Source label bigset (non-strict prefix, so is_relevant doesn't gate on a tech keyword; text is seeded with "hiring software engineers" for scoring anyway).
• Domain resolution (the crux): careers URL on the company's own domain → used directly. ATS hosts (lever.co / greenhouse.io / ashbyhq.com / recruitee / bamboohr / workday / …) → company slug pulled from the URL → MX-checked <slug>.{com,io,co,ai}, then company-name guesses. The resolved domain becomes apply_url so email_finder resolves emails on the company's own domain — ATS domains are in JOBBOARD_DOMAINS and would otherwise be rejected.
• Idempotent: _append_review_queue dedups on apply_url, so re-dropping the same CSV never double-queues.

Generate the CSV: build a dataset in Bigset with a prompt like "Startups that raised funding in the last 30 days, with what they do, funding stage, location, and careers page", export CSV, drop it at ~/.freelance-hunt/bigset_leads.csv.

Honest limits: Bigset rows carry no recruiter name → email_finder falls back to role-account emails (careers@/hiring@) that verify_picks prunes on non-Workspace domains. Strong company source, weak person source — yield is gated by those pre-existing pipeline constraints, not the fetcher.

preflight (run first, before any send)

Verify SPF/DKIM/DMARC on sending domain. Gmail spam-buckets unauthenticated bulk.

bash ~/.claude/skills/hunt/scripts/check_dns.sh rohitraj.tech

scan

Fetch fresh jobs, score against project DB, output ranked top-10.

python3 ~/.claude/skills/hunt/scripts/scan.py

Output: ~/.freelance-hunt/today_picks.json — 10 jobs with email + score + matched projects + pitch variant.

send

Approval gate → fire emails via Gmail SMTP. Cap 95/24h. Throttle 45-60s + jitter.

python3 ~/.claude/skills/hunt/scripts/send.py            # serial render+send (~150min for 95 picks)
python3 ~/.claude/skills/hunt/scripts/parallel_send.py   # parallel render + serial send (Recommended, ~85min)

Reads today_picks.json, prints all drafts, waits for y confirm. Logs sends to ~/.freelance-hunt/sent.json.

verify-picks (Default in auto.py, added 2026-05-17)

Pre-send email validator. Filters today_picks.json BEFORE parallel_send.py to prevent bounces. Pure DNS path — no API calls, no quota, ~4s for 100 picks.

python3 ~/.claude/skills/hunt/scripts/verify_picks.py [--workers=16] [--strict]

Filters:

1. Cross-check blocklist.json (bounces auto-added by learn_replies.py)
2. DNS MX lookup via dnspython — drop domains without mail server
3. Hardcoded disposable-domain list (60+ providers)
4. Heuristic: drop role_account (hiring@/jobs@/careers@/hr@/etc) on domains WITHOUT Google/Microsoft Workspace MX. Small companies without Workspace rarely create hiring@ mailbox → top bounce source.
5. --strict also rejects role_account on Workspace MX (very conservative)

company-dedup (Default in auto.py, added 2026-06-14) — company-level re-pitch gate

Pre-send filter that runs right after verify_picks, before parallel_send. The email deduper (scan.load_seen_emails + dedup_window.json) dedupes by EXACT email, but sent.json has no company field — so careers@acme.com and jason@acme.com are two different leads and the same company gets pitched twice. This drops any pick whose domain was already pitched within the dedup window.

python3 ~/.claude/skills/hunt/scripts/company_dedup.py stats            # top companies by # pitches (≥2 = re-pitched)
python3 ~/.claude/skills/hunt/scripts/company_dedup.py check            # report re-pitches in today_picks.json (exit 1 if any)
python3 ~/.claude/skills/hunt/scripts/company_dedup.py check --in-place # pipeline form: drop them from today_picks.json (.bak backup)

• Excludes free-mail (gmail/outlook/…) and multi-tenant ATS/newsletter domains (greenhouse/lever/substack/…) — one such domain ≠ one company, so collapsing them would wrongly suppress (esp. cofounder gmail leads). Tune the EXCLUDE set in the script if an aggregator (e.g. arc.dev, japan-dev.com) shows up as a false "re-pitch" in stats.
• Window defaults to the active email dedup window (dedup_window.json, else 60d); override with --window N.
• Non-fatal in auto.py — a filter error never blocks the send.

Cache: ~/.freelance-hunt/verify_cache.json — avoids re-resolving same email. Caches facts only (MX hosts + API verdict), never the keep/drop decision — that policy is recomputed every call (2026-06-06 lesson: a cached verdict silently defeats --strict).

mailcheck.ai abandoned 2026-05-17 (IP rate-banned at 429 after first burst). DNS-only path = no quota, scales to thousands/run.

HTTPS email-verification (optional, added 2026-06-11) — the volume unlock. Port 25 is blocked here, so email_finder can't SMTP-verify and emits best-guess hiring@/recruiting@<domain> addresses; the bounce-storm incidents forced --strict to blanket-drop them all (a typical run: 351 candidates → 2 sent). Set a verifier key and verify_picks will instead confirm each guessed mailbox over HTTPS and KEEP the deliverable ones:

• Env (in .env, all optional; no-op until EMAIL_VERIFY_API_KEY is set — behaviour byte-identical to DNS-only without it):

  EMAIL_VERIFY_PROVIDER=reoon        # reoon | neverbounce | zerobounce
  EMAIL_VERIFY_API_KEY=...           # free-tier key, no port 25 needed
  EMAIL_VERIFY_MAX=120               # per-run call cap (protects free quota; verdicts cached across runs)
  EMAIL_VERIFY_TIMEOUT=12
  

• Scope — consulted ONLY for picks we'd otherwise drop as a right-audience role-account guess (role_no_workspace/role_strict, incl. recruit*/talentacq* via ROLE_ISH_SUBSTRINGS). valid→keep (reason=api_verified), invalid→drop (api_invalid), unknown/catch-all/error/over-cap→heuristic fallback. Never called for junk/role_substring wrong-audience drops (a deliverable legal@/investors@ is still the wrong audience), no_mx, or already-kept picks — so credits buy only net-new deliverable volume.
• Fail-safe: any API error/timeout/over-cap → unknown → the existing heuristic drop stands, so a flaky verifier can never cause a bounce.

parallel-send (Default in auto.py, added 2026-05-16)

LLM render bottleneck: serial send.py calls LLM per pick → 60min for 95 picks. parallel_send.py uses multiprocessing.Pool(4) to fork 4 worker processes that render in parallel — preview phase 60min → ~8min. SMTP send loop still serial (Gmail rate-limits parallel connections from same account).

python3 ~/.claude/skills/hunt/scripts/parallel_send.py [--workers=4] [--dry-run] [--no-yes]

Pattern: always parallelize I/O-bound work. Render = LLM API (I/O). MX lookup = DNS (I/O). SMTP probe = TCP (I/O). All forkable via multiprocessing.Pool or ThreadPoolExecutor. Codified for daily use — auto.py invokes parallel_send.py by default, never raw send.py.

State: ~/.freelance-hunt/rendered_drafts.json (pre-rendered drafts, resumable if SMTP fails). Log: ~/.freelance-hunt/parallel_send.log.

followup

T+4d bump + T+10d breakup pitches against unanswered prior sends. Reads sent.json, finds entries at right age that haven't been replied/declined, fires same SMTP path.

python3 ~/.claude/skills/hunt/scripts/followup.py
python3 ~/.claude/skills/hunt/scripts/followup.py --dry-run

Mark a thread as replied to stop follow-ups: edit sent.json and set "replied": true on the matching entry. Same for "declined": true.

build-projects

One-time scrape rohitraj.tech → ~/.freelance-hunt/projects.json. Re-run monthly.

python3 ~/.claude/skills/hunt/scripts/build_projects.py

apply-learnings (auto runs at start of auto.py, added 2026-05-18)

Parses ~/wiki/wiki/project-freelance-hunt.md + ~/.claude/projects/.../memory/freelance_hunt_*.md for documented bad sources / blocked emails / blocked domains. Idempotently adds to disabled_sources.json + blocklist.json. The wiki becomes truth-source: any decision captured there ("disable japan-dev", "block @forwarder-domain.com") propagates to runtime on next auto.py run — no manual state-file edits.

Recognition rules:

• Markdown table rows under a ## Known bad sources (or similar) heading → first-column labels treated as disabled sources.
• \source-label` in prose with disable-hints (dead, disabled, permanently disabled, auto-cut, etc.) within the same line/sentence → disabled. Anti-hints (dependency, import, package`) suppress (so dnspython etc. don't get flagged).
• Emails under an ## Email-level blocklist (or similar) heading → unconditionally blocklisted.
• @domain.com mentions near block-hints (bounce, forwarder, fake, ada) → added as domain-block entry.

python3 ~/.claude/skills/hunt/scripts/apply_learnings.py [--dry-run]

State: ~/.freelance-hunt/apply_learnings.log (append-only history), ~/.freelance-hunt/learnings_applied.json (last-applied digest for diff).

Skip in auto.py: python3 auto.py --no-learn.

status

Show daily count, sent log, replies (manual mark for now).

cat ~/.freelance-hunt/sent.json | jq '.[-20:]'

learn-failures (auto runs at start of auto.py, after auto-cut)

Parses ~/.freelance-hunt/auto.log for repeated HTTP errors per source (404, 403/limit, timeouts). Any source failing ≥3 days within last 7d is appended to disabled_sources.json. Complements auto-cut which gates by email-yield, not transport health. scan.py honors disabled list at fetch-call time (early-return []).

python3 ~/.claude/skills/hunt/scripts/learn_failures.py [--days=7] [--threshold=3] [--dry-run]

State: ~/.freelance-hunt/learn_failures.log (append-only decisions).

auto-tune-dedup (auto runs at start of auto.py)

Auto-shrinks/expands the email dedup window based on yesterday's send count vs MIN_DAILY_TARGET. Bounds 14d–90d, step ±10d. Writes ~/.freelance-hunt/dedup_window.json consumed by scan.load_seen_emails() and enrich_apify.load_seen_emails().

python3 ~/.claude/skills/hunt/scripts/auto_tune_dedup.py [--target=N] [--dry-run]

State: ~/.freelance-hunt/auto_tune_dedup.log.

learn-replies (auto runs at start of auto.py)

Connects to Gmail INBOX via IMAP (GMAIL_USER + GMAIL_APP_PASSWORD), scans last 30d for replies/bounces/auto-replies matching sent.json recipients. Marks each sent.json entry with replied/reply_date/reply_subject/reply_snippet, bounced/bounce_reason, or autoreply flags. Bounced emails auto-appended to blocklist.json. Per-source 30d reply+bounce rate written to ~/.freelance-hunt/source_reply_rate.json.

Filters: classifies by From (mailer-daemon/postmaster → bounce), Subject ("delivery status notification" → bounce), Subject/body ("out of office" / "automatic reply" → autoreply), else → genuine reply.

python3 ~/.claude/skills/hunt/scripts/learn_replies.py [--days=30] [--dry-run]

State: ~/.freelance-hunt/learn_replies.log, ~/.freelance-hunt/source_reply_rate.json.

auto-cut (auto runs at start of auto.py)

Reads ~/.freelance-hunt/source_health.json rolling stats. Any source with dead_streak ≥ 3d AND raw ≥ 5 gets appended to ~/.freelance-hunt/disabled_sources.json. scan.py filters items from disabled sources at write time. Decisions logged to ~/.freelance-hunt/auto_cut.log.

python3 ~/.claude/skills/hunt/scripts/auto_cut.py             # apply cuts
python3 ~/.claude/skills/hunt/scripts/auto_cut.py --dry-run   # report only

discover-sources (auto runs at start of auto.py)

Daily research for new sources. Multi-strategy discovery:

• 4 curated awesome-list READMEs (lukasz-madon/awesome-remote-job, tramcar/awesome-job-boards, remoteintech/remote-jobs, poteto/hiring-without-whiteboards)
• HN Algolia search — queries like "best remote job board", "freelance platform 2026", "post job free" → community-recommended boards
• Reddit searches in r/cscareerquestions, r/digitalnomad, r/forhire — "where to post" / "remote job site" threads
• Curated PROMOTE_SEED — 10 known high-yield boards (hasjob, wellfound, cutshort, hirect, outerjoin, remoteleaf, echojobs, nodesk, europeremotely, remoteok.io) auto-promoted on first run, skip 3-trial ramp

Extracts ~750-800 unique candidate domains. Tests 15 unseen domains/day for email yield. Promotes any candidate to active after 3 trial wins (≥1 email per fetch). Promoted domains flow into scan via generic_fetch.py next run.

python3 ~/.claude/skills/hunt/scripts/discover_sources.py            # full run
python3 ~/.claude/skills/hunt/scripts/discover_sources.py --max=20   # cap tests
python3 ~/.claude/skills/hunt/scripts/discover_sources.py --dry-run  # log only

State: ~/.freelance-hunt/source_candidates.json (per-domain trial/win counts + promoted flag). Daily report: ~/wiki/raw/source-discovery-YYYY-MM-DD.md.

top-up (auto runs after first send)

Loops scan → enrich → send up to 5 times until MIN_DAILY_TARGET (default 80) hunt emails sent today. Each loop widens HN reach via DEEP_SCAN_MONTHS env (3 → 4 → 5 → 6 → 7 months back). Tolerates 1 zero-delta loop; bails after 2 consecutive zero-delta loops (supply truly exhausted) or target reached. Note: Gmail SMTP cold-send ceiling ~50-100/day before spam flagging — domain SPF/DKIM/DMARC must be clean.

python3 ~/.claude/skills/hunt/scripts/top_up.py                 # default target 80
python3 ~/.claude/skills/hunt/scripts/top_up.py --target=100 --max-loops=7

State: ~/.freelance-hunt/top_up.log. Env: MIN_DAILY_TARGET=80 in .env.

wiki-dump (auto runs at end of auto.py)

Mines today's run + 7d rolling source health and writes a Karpathy-pattern raw source to ~/wiki/raw/freelance-hunt-YYYY-MM-DD.md. Sections:

• Pipeline summary (sent today, picks, queues)
• Top stacks / problems / roles / domains / seniority / locations / engagement / salaries (regex token freq from JD text)
• Source efficiency — today (raw / emails-found / sent per source)
• Source efficiency — 7d rolling (send-rate %, dead-streak days, OK/ZERO/DEAD flag)
• Suggested cuts — auto-flags sources with ≥3d zero-email streak + ≥5 raw items
• Auto-decisions today — disabled sources, promoted sources from log
• Recruiter hiring patterns — regex extraction of "looking for X", "must have Y", "experience with Z", "N+ years of …" phrases grouped by pattern
• Score distribution histogram, top-5 picks with quotes, sent log, top-5 cofounder leads

Final stdout prints DAILY SUMMARY + MARKET ANALYTICS box: top stack / domain / role / seniority / engagement / location + 3 salary samples + sent counts + dead source flags.

Pipeline writes ONLY to ~/wiki/raw/. Run "compile my wiki" to merge into ~/wiki/wiki/project-freelance-hunt.md + rolling trends-*.md (LLM-driven per ~/wiki/CLAUDE.md).

State: ~/.freelance-hunt/source_health.json keeps rolling 7d per-source stats.

python3 ~/.claude/skills/hunt/scripts/wiki_dump.py             # write today's raw file
python3 ~/.claude/skills/hunt/scripts/wiki_dump.py --date=2026-05-08

wiki-compile (auto runs after wiki-dump)

After wiki_dump.py, auto.py shells out to a headless Claude Code call to merge raw → compiled wiki:

claude --add-dir ~/wiki --permission-mode acceptEdits -p "compile my wiki"

Flag rationale:

• --add-dir ~/wiki — ~/wiki/ lives outside auto.py cwd; without this the headless session denies tool access.
• --permission-mode acceptEdits — non-interactive -p mode can't approve Write/Edit prompts; auto-accepts edits but still gates Bash. Safer than --dangerously-skip-permissions.

Disable: AUTO_COMPILE_WIKI=0 env. Override binary: CLAUDE_BIN=/path/to/claude. Hard timeout 10min — pipeline continues if compile times out or claude CLI not found.

Known noise (cosmetic, doesn't fail rc): SessionEnd hooks from vercel + codex plugins log SyntaxError: Unexpected token '?' — older Node parser hits ?? operator on session shutdown.

State files

• ~/.freelance-hunt/projects.json — project DB (name, stack, summary, url)
• ~/.freelance-hunt/sent.json — log of sent emails (recipient, subject, date, score)
• ~/.freelance-hunt/today_picks.json — current run's picks
• ~/.freelance-hunt/blocklist.json — emails to never contact
• ~/.freelance-hunt/source_health.json — rolling 7d per-source stats (raw, emails, sent, dead-streak)
• ~/.freelance-hunt/disabled_sources.json — auto-cut list (≥3d dead, ≥5 raw) + learn_failures cuts
• ~/.freelance-hunt/source_failures.json — per-source failure ledger (date list + last_error)
• ~/.freelance-hunt/dedup_window.json — auto-tuned dedup window (days, reason, updated)
• ~/.freelance-hunt/demand_snapshot.json — last-7d top stacks/domains, injected into pitch STACK_SNAPSHOT
• ~/.freelance-hunt/source_reply_rate.json — per-source 30d reply + bounce rate
• ~/.freelance-hunt/source_candidates.json — discovery trial state (domain → trials/wins/promoted)
• ~/.freelance-hunt/auto_cut.log — append-only auto-cut decisions
• ~/.freelance-hunt/discover.log — append-only discovery runs
• ~/.freelance-hunt/email_finder_cache.json — MX lookups + verified/rejected emails + catchall-domain flags (DIY finder, added 2026-05-16)
• ~/.freelance-hunt/email_finder.log — append-only emfind runs (processed/found/promoted counts)
• ~/.freelance-hunt/apify_review.json — review queue (legacy filename; now consumed by email_finder.py, written by scan.py + adzuna_fetch.py + bigset_fetch.py)
• ~/.freelance-hunt/firecrawl_review.json — Firecrawl review queue
• ~/.freelance-hunt/bigset_leads.csv (or bigset/*.csv) — Bigset CSV export drop-zone, read by bigset_fetch.py (input; you create it)
• ~/.freelance-hunt/bigset_fetch.log — append-only bigset ingest runs (parsed/queued/dropped counts)
• ~/.freelance-hunt/.env — GMAIL_USER, GMAIL_APP_PASSWORD, ADZUNA_APP_ID, ADZUNA_APP_KEY, FIRECRAWL_API_KEY
• ~/wiki/raw/freelance-hunt-YYYY-MM-DD.md — daily Karpathy-wiki source (auto-written)

Config (.env)

GMAIL_USER=rohitgupta2432@gmail.com
GMAIL_APP_PASSWORD=<in ~/.freelance-hunt/.env only — never commit the real value here>
# DAILY_CAP and COFOUNDER_CAP removed 2026-05-06 — only Gmail SMTP rolling-24h hard cap (~95) still gates sends.
ROLLING_24H_CAP=95
THROTTLE_MIN=45
THROTTLE_MAX=60
SENDER_NAME=Rohit Raj
SITE_URL=https://rohitraj.tech
GITHUB=rohitguta2432
TIMEZONE=flexible across timezones
OVERLAP=5+ hrs overlap any TZ
HOURS_PER_WEEK=30-40

# Apify integration — REMOVED 2026-05-16 (credits exhausted, no replenish).
# Scripts archived as apify_fetch.py.removed-* and enrich_apify.py.removed-*.
# Env vars commented as APIFY_*_REMOVED_20260516=...

# Adzuna API — RESTORED 2026-05-16 (native fetch, no Apify wrapper, free 1000 calls/mo)
ADZUNA_APP_ID=xxx                                  # developer.adzuna.com free signup
ADZUNA_APP_KEY=xxx
ADZUNA_COUNTRIES=us,gb,in,au,ca,de,fr,nl,sg,za,nz,pl,br,it,es,mx,ch,at,ru
ADZUNA_QUERY=developer                             # single keyword; OR syntax broken in API
ADZUNA_CATEGORY=it-jobs                            # filters out medical/retail tech roles
ADZUNA_MAX_PAGES=2                                 # 50/page = 100/country/run
ADZUNA_MAX_DAYS_OLD=30                             # 0 disables date filter

# Firecrawl integration (optional)
# Free tier: 500 credits/mo. Each board scrape ~5-15 credits.
FIRECRAWL_API_KEY=fc-xxx
FIRECRAWL_QUERY=python+developer+remote
FIRECRAWL_LOCATION=Remote
FIRECRAWL_MAX_CREDITS=100
FIRECRAWL_SOURCES=weworkremotely,wwr-backend,wwr-fullstack,arc

# Daily target — top_up.py loops scan+enrich+send until reached
MIN_DAILY_TARGET=100                               # bumped 80→100 2026-05-16 (50 hunt + 50 cofounder target)

# Scrapling anti-bot fallback (added 2026-05-31) — see "Scrapling fallback" section.
# All optional; these are the defaults. Set USE_SCRAPLING=0 to hard-disable.
USE_SCRAPLING=1
SCRAPLING_VENV=~/.freelance-hunt/.venv-scrapling
SCRAPLING_TIMEOUT=90                               # per-fetch subprocess timeout (s)
SCRAPLING_MAX_LAUNCHES=60                          # per-process browser-launch cap
# SCRAPLING_PROXY=http://user:pass@host:port       # only for IP-rate-limited sources

Pitch variants

• match — used when ≥1 project from DB scores ≥2 keywords vs job. Lead with matching projects.
• adapt — fallback when 0 match. Polyglot angle.

Templates: ~/.claude/skills/hunt/templates/pitch_match.txt + pitch_adapt.txt.

Scrapling fallback (anti-bot bypass, added 2026-05-31)

Hunt fetches via urllib first. When a source returns a hard block (HTTP 403/429/503/401/406) or a transport error, the fetch auto-escalates to scrapling — curl_cffi Chrome-TLS impersonation first, then headless Chromium (Cloudflare-solving) if that's still blocked. Pure fallback: never on the happy path, so it costs nothing when urllib succeeds, and the whole thing is a no-op unless the venv exists and USE_SCRAPLING is on.

Why: Reddit hard-blocks its .json API for datacenter traffic — 403 even through a real browser — which had killed the entire cofounder phase ("0 sent", 2026-05-29). old.reddit.com HTML still renders via the stealth browser, so the Reddit fetchers fall back to parsing that.

Install (one-time, isolated)

scrapling[all] pulls Playwright + Chromium, so it lives in its OWN venv — deliberately NOT in the hunt runtime python (keeps the heavy browser stack off the scan hot path):

python3.12 -m venv ~/.freelance-hunt/.venv-scrapling
~/.freelance-hunt/.venv-scrapling/bin/pip install "scrapling[all]>=0.4.8"
~/.freelance-hunt/.venv-scrapling/bin/scrapling install --force   # downloads Chromium

Architecture (3 new scripts)

• _scrapling_worker.py — the ONLY file that imports scrapling. Runs under the venv interpreter; fetches one URL, writes the raw body to stdout. Modes: get (curl_cffi, raw bytes — JSON/RSS parse unchanged), stealth (browser + solve_cloudflare — walled HTML), auto (get → escalate on block/empty).
• stealth_fetch.py — imported by hunt's runtime python (stdlib only). Shells out to the worker under the venv interpreter. stealth_get(url, mode) returns the body or None (never raises). Enforces a per-process browser-launch budget.
• reddit_fetch.py — old.reddit listing parser + targeted post-body fetch. Cofounder leads need an email (which lives in post bodies, not listings), so it spends one body fetch per title-matched post — bounded by the launch budget.

Wired into: scan.py fetch_url() (covers every RSS/JSON source that routes through it), scan.py fetch_reddit_subs(), and cofounder.py reddit_search() + reddit_query_search().

Env (all optional; defaults are sane — added to .env section below)

USE_SCRAPLING=1                                    # master switch (0 = hard-disable)
SCRAPLING_VENV=~/.freelance-hunt/.venv-scrapling   # venv dir
SCRAPLING_TIMEOUT=90                               # per-fetch subprocess timeout (s)
SCRAPLING_MAX_LAUNCHES=60                          # per-process browser-launch cap (bounds wall-clock)
SCRAPLING_PROXY=                                   # http://user:pass@host:port for IP-rate-limited sources

Honest limits (don't oversell this)

• Restores access + title-matching, not magic. Reddit cofounder posts rarely expose a raw email (they say "DM me"), and the pipeline hard-requires an email — so lead yield stays gated by that pre-existing constraint. The net win is resilience (no more 403 death of the whole phase) + reachability of the posts that do carry an email, plus restored title-signal matching.
• The old disabled feeds (cryptojobslist, web3.career, aijobs, pangian) are 404 — dead/moved URLs, not anti-bot walls (re-verified 2026-05-31; the wiki's "Cloudflare 403" note is stale). Scrapling can't revive a 404, so they stay in disabled_sources.json. fetch_url only escalates on 403/429/503, not 404 — so no browser launch is wasted on them.
• IP-based blocks (as opposed to UA/TLS-fingerprint) need a residential proxy via SCRAPLING_PROXY; the stealth browser alone won't beat a datacenter-IP ban.
• Test the bridge manually: python3 ~/.claude/skills/hunt/scripts/stealth_fetch.py <url> stealth.

Sources

Hunt (scan.py)

Parallel scan (2026-07-13): all sources now fetch concurrently via a ThreadPoolExecutor(8) registry in scan.py main() (was sequential, 8-9 min → ≈ slowest single source). Registry tags skip auto-disabled sources at fetch time; Reddit has a circuit breaker (2 consecutive blocked subs → skip the rest that run).

Order (Apify removed 2026-05-16):

1. HN/Reddit/Jobicy/Himalayas/RemoteOK — inline-email sources (highest yield)
2. WWR/Remotive/Arbeitnow/Builtin/4DayWeek/SkipTheDrive — RSS (low email yield, → review queue)
3. Generic (auto-promoted) — discover_sources.py promoted boards
4. Adzuna API (native fetch — 19 countries, free 1000 calls/mo) — items → review queue
5. Firecrawl — paid LLM scrape on WWR/arc.dev → review queue
6. Bigset CSV (funded-startup leads via bigset_fetch.py; runs in auto.py run_hunt after scan, not inside scan.py) — items → review queue

Active sources:

• HN "Who is Hiring" via Algolia API (current month + DEEP_SCAN_MONTHS deeper)
• HN "Freelancer? Seeking freelancer?" via Algolia
• hnhiring.com (URL format fixed 2026-05-16: /<month>-<YYYY>)
• RemoteOK API (https://remoteok.com/api)
• Reddit subs: r/forhire, r/jobbit, r/remotejs, r/hireaprogrammer, r/freelance_forhire
• WeWorkRemotely RSS (/categories/remote-programming-jobs.rss)
• Remotive API (/api/remote-jobs?category=software-dev)
• Jobicy API (https://jobicy.com/api/v2/remote-jobs)
• Himalayas API (https://himalayas.app/jobs/api)
• Working Nomads (https://www.workingnomads.com/api/exposed_jobs/)
• Arbeitnow (https://www.arbeitnow.com/api/job-board-api) — EU/visa angle
• Justjoin.it API (https://api.justjoin.it/v2/user-panel/offers — Version: 2 header; fixed 2026-05-16)
• Landing.jobs, NoFluffJobs, BuiltIn
• 4DayWeek.io RSS, SkipTheDrive RSS
• Adzuna API (19 countries: gb/us/in/au/ca/de/fr/nl/sg/za/nz/pl/br/it/es/mx/ch/at/ru)
• japan-dev.com (Japan, English-speaking tech jobs, ~hundreds, inline @japan-dev.com forwarder emails)
• wanted.co.kr (Korea, JSON API, ~100/run, no inline email → review queue)
• Bigset CSV (funded-startup leads — bigset_fetch.py ingests a Bigset export → review queue; ATS careers URLs resolved to the company's own mail domain; runs in run_hunt)
• Freelancer.com (added 2026-07-13 — public no-key API api/projects/0.1/projects/active/?query=, 10 stack queries → freelancer_com_projects.json bid queue; in-platform contact, never emailed)
• YC jobs (added 2026-07-13 — ycombinator.com/jobs server-rendered HTML → yc_jobs.json apply queue; founding-engineer roles ranked first; apply via Work at a Startup)

Action email (added 2026-07-13): action_email.py mails the merged projects/founding-roles view (freelance_projects_view) to GMAIL_USER after each non-dry run — the review-gated leads are the highest-conversion output and were previously buried in state files. Disable: ACTION_EMAIL=0.

FREE-ONLY MODE (2026-07-13, user decision — do not add paid APIs): reoon + Firecrawl keys disabled in .env (both accounts were at 0 credits; user chose free-only over recharging). Consequences: email channel = real found-on-page emails only (free MX heuristic); role-account guesses (hiring@…) drop by design — no verifier means they're bounce risks. The PRIMARY output is the free bid/apply queue (freelancer.com API + freelancermap + peopleperhour + YC founding roles) mailed daily by action_email.py. verify_picks.py still fails LOUD if a provider key is ever set but erroring.

Permanently disabled (URL dead/Cloudflare-walled, 2026-05-16):

• cryptojobslist (Cloudflare 403)
• aijobs.net (no feed exists)
• web3.career (all paths 404)
• pangian.com (404)
• Pangian RSS — global remote
• Justjoin.it API — PL/EU tech
• Landing.jobs Atom feed — EU tech
• NoFluffJobs JSON API — PL/EU tech
• BuiltIn RSS — US city tech (dev-engineering + remote feeds)
• Firecrawl (optional, env-gated via FIRECRAWL_API_KEY) — LLM-powered scrape, JSON-extract via prompt. 10x more efficient per credit than Apify for job boards (tested 2026-05-05: Indeed returned 15/15 relevant Python jobs at 9 credits, vs Apify 100 items mostly janitorial). Free tier 500 credits/mo = 55 board scrapes. Sources: indeed, wellfound, weworkremotely, remoteok, ycombinator. Items WITHOUT email → `/.freelance-hunt/firecrawl_review.json. Daily usage tracked at ~/.freelance-hunt/firecrawl_usage.json. Capped via FIRECRAWL_MAX_CREDITS`.
• Apify Actors (optional, env-gated via APIFY_TOKEN) — Upwork, LinkedIn, Wellfound, Indeed via Actor IDs in env. Job boards strip emails → most Apify items go to manual-review queue at ~/.freelance-hunt/apify_review.json (apply via URL). Items WITH email auto-flow into SMTP pipeline. Costs ~$0.0001–$0.015 per item. Cap via APIFY_MAX_ITEMS (some Actors ignore — verify per-Actor). Skips silently when token unset.

Apify caveats

• Indeed Actor (valig/indeed-jobs-scraper) returns generic feed; position filter often ignored. Use APIFY_INDEED_COUNTRY=us|uk|in|... for region.
• LinkedIn / Wellfound Actors require subscription tier in some cases (403 on free tier).
• Per-Actor input shapes differ — see ACTOR_INPUT_ADAPTERS in apify_fetch.py. Add adapters for new Actors there.
• Manual review file: cat ~/.freelance-hunt/apify_review.json | jq '.[].apply_url' then apply manually.

Cofounder (cofounder.py)

• HN Algolia search — 16 query variants (looking-for-CTO / cofounder / founding engineer / technical partner / equity-for-engineer / etc.)
  • Comments (tags=comment) AND stories+Show HN (tags=story) both scanned — comment-only missed founders posting standalone "looking for CTO" stories.
• Reddit cofounder subs (14): r/cofounder, r/startups, r/Entrepreneur, r/EntrepreneurRideAlong, r/SaaS, r/ycombinator, r/indiebiz, r/SideProject, r/IMadeThis, r/roastmystartup, r/smallbusiness, r/AlphaandBetausers, r/startup, r/indiehackers
  • Two passes per sub: (a) /new.json recent fetch, (b) per-sub search via /search.json?q=<query>&restrict_sr=on for 6 query variants — finds older relevant posts /new misses.
• Indie Hackers RSS (best-effort; IH rarely exposes raw emails)
• BetaList feeds (best-effort; feed URLs return 404 since 2026-05; left wired in case they restore)
• ProductHunt feeds (best-effort; newest launches)
• Stale re-engage: load_sent() only blocks emails sent within last 45d (RECENT_SEND_WINDOW_DAYS). Older sends fall out of the dedup window so cofounder pipeline can re-pitch with refreshed project list. Critical for survival — HN cofounder threads have ~30-active-poster ceiling at any time, full dedup exhausts in ~7d.
• Out-of-scope (gated/JS-rendered, no public API):
  • YC Co-Founder Matching (startupschool.org) — needs login + Playwright
  • Wellfound (AngelList) cofounder listings — no email until apply-flow
  • X / Twitter "looking for technical cofounder" — API gated

Efficiency review — 2026-07-06 (read before "why is volume low?")

Funnel that day: 385 candidates → verify kept 19 (strict) → company_dedup → 4 sent vs target 100. Root causes found + fixed:

1. emfind dead-item loop (FIXED) — email_finder.py pending filter didn't exclude terminal exhausted/no_domain statuses, so the first 500 queue slots were the same failed items re-tried every run (`e

…(truncated)