# Net Rotating Proxies

> Use when a download, scrape, or API pull is blocked or rate-limited by the target (HTTP 429, IP ban, geoblock) and must be routed through proxies, or when fetching many items from a host that throttles per IP (bulk YouTube transcripts, scraping, API harvesting).

- Skill: `bitranox/net-rotating-proxies` (Agent Skill, multi-file: 7 files)
- Install (CLI): `npx skillmds@latest add bitranox/net-rotating-proxies`
- Raw SKILL.md: https://api.skillmd.com/api/skills/bitranox/net-rotating-proxies/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Integrations & APIs
- Author: bitranox (https://skillmd.com/u/bitranox)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/bitranox/net-rotating-proxies

---


# Rotating proxies for blocked / rate-limited fetches

When the local IP is rate-limited or blocked (HTTP 429, ban, geoblock) and you must keep
fetching, route requests through a rotating pool of proxies. Free proxies are flaky and
short-lived, so the method is: keep a persistent pool, re-test it every run, fetch in parallel
(most proxies are slow), prefer proxies that have actually worked, and keep refreshing the pool
in the background while downloading.

## When to use

- A bulk fetch starts returning 429 / "Too Many Requests", or the host bans your IP partway through.
- Client-side fixes do NOT help: changing user-agent, impersonation, or a JS runtime will not beat
  a per-IP quota. Only a different egress IP (a proxy) will. Verify the block is per-IP first.
- You need to pull many items from one throttling host (e.g. YouTube auto-caption transcripts).

## The rules (always)

1. **Persistent pool, disproven at use time.** Proxies constantly die, recover, and appear, so a
   saved list is never trusted as-is. Keep `pool.txt` (all candidates) and `live.txt` (passed a
   reachability test at some point). Note what `validate` actually does: it tests only
   `pool - live - bad`, so an entry already in `live.txt` is NOT re-tested on a later run - the file
   accumulates. Freshness comes from the other end instead: a proxy that fails during a job is
   banned to `bad.txt`, and every reader uses `live - bad`. So treat a live entry as
   not-yet-disproven rather than proven-good, and let the ban path do the pruning.
2. **Discover from public lists**, merge grow-only and deduped into the pool (github raw proxy
   lists, proxyscrape API). Expect thousands of candidates; only a few percent will be usable.
3. **Validate reachability in parallel** against a cheap endpoint on the target host before using
   a proxy for real work, and **record each proxy's latency** (response time of that test) so
   selection can favour fast proxies. **Right-size to the job:** pass `validate --need N` to stop as
   soon as N live proxies are found (and cancel the rest) instead of testing the whole pool - a small
   job needs only a few. Size with a generous `~100%` margin: `N ~= 2 x concurrency` (e.g. 8-wide is
   `--need ~16`, not thousands). The over-provision is what lets the speed-weighted pick run the load on
   the FASTEST proxies while the slower half idles as warm backup (effectively phased out - rarely
   picked). That warm backup is the ONLY instant replacement: a proxy that dies is banned and the
   next pick skips it immediately. The background refresh is what REFILLS the backup, and it is
   slow - with `--background-discovery` it discovers and tops up every 10 minutes (hardcoded, and
   the first pass is at t+10min, not t+0); without that flag it never runs at all. Size `--need`
   to survive ten minutes of attrition rather than to be rescued from it. Distribute requests
   across the pool so no single exit-IP is hammered.
4. **Download in parallel, wide.** Most free proxies are awfully slow (tens of seconds each), so
   parallelism is the only way to make progress: 8 workers minimum, 16 when the pool is large.
5. **Keep good/bad lists and weight by speed, but ROTATE so no proxy is hammered.** A proxy that
   completes a real download goes in `good.txt` (higher base weight); one that fails at the
   connection level goes in `bad.txt` and is excluded. Within the remaining pool, select proxies by
   **weighted random sampling where the weight rises as latency falls**, so faster proxies are used
   far more often than slow ones (the stored validation latency drives this; unknown-latency proxies
   default to the median). Layer a **cool-down / least-recently-used rest** on top of the speed
   weighting: a proxy used within the last `--cooldown` seconds is held out of the pick so the single
   fastest exit-IP is not hit back-to-back; load spreads across the fast half of the pool and the
   cool-down relaxes (oldest-rested first) if it would otherwise starve the pick. Rotate to the next
   proxy on any per-item failure.
6. **Self-optimize in the background: benchmark, swap up, and evict flaky.** While the workers run,
   a background loop keeps the working set = the N FASTEST healthy proxies:
   - discovery + revalidation tops `live.txt` back up so dead proxies get replaced without stopping
     the job (right-sized to `--need`);
   - a benchmark pass re-times the in-pool proxies and trials fresh candidates; when a fresh
     candidate is faster than the slowest **idle** in-pool proxy (never one mid-request), it swaps
     the slow one out for the fast one (`--bench-interval` controls the cadence);
   - per-proxy success/failure is tracked, and a proxy that fails intermittently past
     `--flaky-fail-ratio` is evicted and replaced just like a hard-dead one - so steady state is the
     N fastest, still-functioning proxies.
7. **Resumable.** Make the worklist skip items already done, so a killed run resumes cheaply.

## Tool

`scripts/proxy_pool.py` (HTTP via `httpx2`, otherwise stdlib; cross-platform) implements all of
the above. Run it with `uv run` so the `httpx2` dependency (declared in the script's PEP-723
header) is fetched into an isolated env. Subcommands:

    # 1. build the pool (grow-only, deduped)
    uv run scripts/proxy_pool.py --store ./.proxies discover

    # 2. test reachability against the target host, grow live.txt (parallel)
    uv run scripts/proxy_pool.py --store ./.proxies validate \
        --test-url https://www.youtube.com/generate_204 --workers 150 --need 10   # stop at 10 live (right-size)

    # 3. run a worklist through rotating proxies, 16-wide, refreshing the pool in the background
    uv run scripts/proxy_pool.py --store ./.proxies run \
        --worklist items.txt --workers 16 --per-item-proxies 8 --need 24 --background-discovery \
        --test-url https://www.youtube.com/generate_204 \
        --success-glob 'out/{item}*.vtt' \
        --cmd 'yt-dlp --proxy http://{proxy} --skip-download --write-auto-subs --sub-langs en.*,en --sub-format vtt -o out/{item}.%(ext)s https://www.youtube.com/watch?v={item}'

`{proxy}` (host:port) and `{item}` are substituted per attempt. The `--cmd` is parsed once with
`shlex.split` and run as an argv list with NO shell (OS-independent, injection-safe), so it must
be a single command: no pipes, `||`, `$?`, redirects, or `case`. It runs once per proxy until one
succeeds.

`run` holds a self-optimizing working set of the `--need` fastest healthy proxies - pass it, or
`--need` defaults to None and the working set is not right-sized at all: it rotates the
pick with a `--cooldown` rest so no exit-IP is hammered, evicts a proxy once its failure fraction
exceeds `--flaky-fail-ratio`, and (with `--background-discovery`) re-benchmarks the pool every
`--bench-interval` seconds, swapping a freshly-found faster proxy in for the slowest idle one.

### How success / failure is decided (no shell glue)

The tool classifies each attempt itself, portably:

- **success** = return code 0 AND, when `--success-glob` is given, a matching output file exists
  (many tools exit 0 without producing output, so prefer `--success-glob`) -> proxy goes in
  `good.txt`, rotation stops for that item.
- **dead proxy** = the command's combined stdout+stderr matches `--dead-regex` (connection
  refused, reset, timeout, unreachable, proxy/tunnel errors) -> proxy goes in `bad.txt`, excluded
  next time.
- **otherwise** (incl. timeout, 429, transient) -> rotate to the next proxy, do not ban it.

Pick `--test-url` and `--success-glob` to match the host and tool you are unblocking; widen
`--dead-regex` if your tool words connection failures differently.

## Notes

- Stay within lawful use. Only fetch content you are authorised to access, and respect the
  target's terms of service and robots directives. Rotating egress IPs spreads a legitimate bulk
  fetch past a per-IP rate limit; it is not for defeating a block meant to keep you out.
- Confirm the block is per-IP before reaching for proxies; if a few requests still succeed from
  your own IP, slow down instead.
- Free-proxy hit rate is low and decays within minutes; the background refresh and good/bad lists
  exist precisely because of this. Re-run `discover`+`validate` between sessions.
- Treat proxied traffic as untrusted transport: only route public, non-sensitive fetches through
  free proxies, never anything authenticated or private.