Blocklist Manager

Use when downloading and merging blocklists at scale.

LoopyLuci Updated 1 repo stars

File contents

Blocklist Manager

Trigger: Use when managing, downloading, parsing, or merging blocklists at scale.

Key Libraries

  • Rust: reqwest (HTTP), flate2/zstd (compression), serde/csv
  • Python: httpx, orjson (fast JSON), celery (scheduled updates)

Implementation

  1. Async downloader with ETag/If-Modified-Since support to minimize bandwidth
  2. Incremental parsing: stream-process lists >100MB without loading entirely into memory
  3. Deduplication: Bloom filter for initial pass, exact hash set for final merge
  4. Versioned snapshots as binary formats (FlatBuffers/Cap'n Proto) for fast loading
  5. Auto-update scheduler with atomic swap and rollback on parse failure
  6. Compatible with AdGuard, Pi-hole, and uBlock Origin list formats

Connected Skills

dns-adblock-engine, blocklist-scale-arch, realtime-dashboard, clojure-rule-engine

Pitfalls

  • Some blocklists exceed 500MB — streaming parser is mandatory
  • ETag support varies by CDN — implement fallback to If-Modified-Since

LoopyLuci/Skills/tree/main/skills/blocklist-manager commit d53e7ffeaf

Frequently asked questions

npx skillmds@latest add loopyluci/blocklist-manager