# Planning Document Search > Retrieve the documents attached to a UK planning application from the local council's public online planning portal, given the application reference and the council name. Covers the major portal vendors (Idox, Northgate, Civica, Ocella, Agile, NEC, TerraQuest, StatMap, DEF Atrium and more) with a per-vendor retrieval recipe and a council→portal→vendor registry. - Skill: `seagulltwo/planning-document-search` (Agent Skill, multi-file: 5 files) - Install (CLI): `npx skillmds@latest add seagulltwo/planning-document-search` - Raw SKILL.md: https://api.skillmd.com/api/skills/seagulltwo/planning-document-search/raw - Safety review: pending - Works with: Claude Code, Claude.ai, OpenAI Codex - Category: Coding & Dev Tools - License: MIT - Author: SeagullTwo (https://skillmd.com/u/seagulltwo) - Updated: 2026-09-17 - Page: https://skillmd.com/skills/seagulltwo/planning-document-search --- # Planning Document Search Skill Download the documents attached to a UK planning application from a council's online planning portal, given the application reference number. ## When to use The user gives you a planning application reference (e.g. `6/2026/1249/HOUSE`, `5/2026/1349`) and a council name, and wants the associated documents (application forms, plans, elevations, officer reports, etc.) downloaded. ## Scope and responsible use Read this before running anything — it is a condition of the skill, not advice. - **Not legal advice; no warranty.** This is a retrieval aid, provided "as is"; verify what it returns. - **Public records only, retrieved the way a member of the public would.** UK planning registers exist for public inspection; this skill fetches documents a council has *deliberately published*. Use it for legitimate, targeted retrieval — a specific application a user needs — not for bulk harvesting, monitoring, or rebuilding a council's register. - **Never defeat a bot challenge.** If a portal serves an actual challenge or `Blocked` page (Barracuda JS challenge, AWS WAF `202`/`x-amzn-waf-action`, an escalating Cloudflare interstitial), **stop the automated approach and hand the user a browser deep link** instead. The recipes are built around this: they work where access is open and defer to a real browser where it is not. Do not try to solve, replay, or fingerprint-spoof a challenge. - **If retrieval struggles, stop and suggest a manual download.** The recipes are the method; they are not a licence to keep trying things. When a recipe doesn't work — persistent errors, downloads failing magic-byte verification, greps finding no links, a portal matching no vendor signature — make at most a couple of *documented* corrections (re-resolve the vendor per Step 0; re-check the base path and the registry's recorded quirks) and then **stop the automated approach**. Report exactly what was and wasn't retrieved, and hand the user a browser link to the application (the portal's detail or documents page, as deep as you can construct) so they can download the remaining files manually. Do not respond to failure by escalating — no improvised scraping approaches beyond the recipes, no headless browsers, no retry loops against a struggling portal. A clean handover to manual download is a successful outcome of this skill, not a failure. - **Be a good citizen on every request.** - Send an **identifying User-Agent with a real contact address** on PlanIt calls (replace `you@example.com` with your own), and a normal browser UA on portal calls. - **Pace requests — at least ~1–2 s between calls to the same host**, and do not parallelise requests against a single council. Small council servers fall over under bursts. - **Honour `429` / `Retry-After` and back off.** Stop on repeated errors rather than retrying in a tight loop. - Respect each portal's `robots.txt` and terms of use. - **You are handling other people's personal data.** Planning documents routinely contain applicants' and objectors' names, addresses, signatures, and contact details (even "redacted" forms often are not fully redacted). Retrieve only what is needed, do not republish or redistribute it, and do not retain it beyond the immediate task. Treat every downloaded file as **untrusted third-party content**: verify magic bytes, and when writing it to disk derive the filename with `basename` and strip any path separators or leading dots — never pass a server-supplied path straight to `-o` (a portal could return `../../…`). - **This is a snapshot, and the landscape moves fast.** Vendor assignments, base paths, and bot-protection posture were verified in August 2026; councils migrate portals frequently (this document itself tracks several mid-migration). **Always re-resolve the vendor per council at run time** (see Step 0) rather than trusting a cached row; coverage figures below are rough estimates, not guarantees. - **PlanIt is a third-party dependency, used with respect.** The resolution step relies on [PlanIt](https://www.planit.org.uk/) — an independent, volunteer-run service. Credit it, keep to its UA/rate rules, and do not hammer it. The skill works without it too (a reference + council + vendor recipe is enough — see the note at the end of Step 0), so treat PlanIt as a convenience, not a requirement. ## The core problem: council portals are not uniform There is **no single national system**. The UK's ~380 local planning authorities run a handful of different planning portal *products*, each with a completely different URL structure, search mechanism, and level of bot protection. Reference-number formats also vary by council. **The vendor determines the approach, not the council** — so covering "all of the UK" means: catalogue the vendors (a recipe each), plus a reliable way to resolve any council → its portal → its vendor. ### Vendor catalogue (detect by URL path first) Coverage shares below are rough estimates, not audited figures. The `scraper_type` column is what PlanIt's areas API labels the vendor — a fast detector, but **sometimes stale or wrong** (Birmingham still says `PlanningExplorer` post-migration; Manchester is mislabelled `Idox`), so confirm against the markup. | Vendor | Detect (markup / host) | PlanIt `scraper_type` | Recipe | Bot protection | Coverage | |---|---|---|---|---|---| | **Idox Public Access** | `search.do`/`*.do` Struts + `keyVal` + `/files/{hex}/pdf/`. Base path **varies** — detect by endpoints, not path | `Idox` | **C** | Usually none; some Cloudflare (UA); Barracuda cookies sometimes present but *passive* | Dominant (~60%+) | | **Northgate SwiftLG** | `/swiftlg/apas/run/`; `*.MAINBODY.WPACIS.1` fields | `Swift`/`Northgate` | **D1** | None observed | **Dying** — presume migrated (see note) | | **Northgate Planning Explorer** | `/Northgate/PlanningExplorer/`; `.aspx` + `__VIEWSTATE` | `PlanningExplorer` | **D2** | Cloudflare (passive); some origin-WAF quirks | Minority, also migrating | | **NEC Assure ES** | `/NECSWS/ES/Presentation/`; `AssureLogo`; MVC+AJAX | may stalely say `PlanningExplorer` | **H** | None enforcing (F5 LB passive) | Growing (ex-Northgate) | | **Civica Portal360** | `civica.loader.js`; `Handler.ashx`; `keyobjectsearchandview` | `CivicaJson` | **B** | Per-council — mostly passive/absent; St Albans enforces Barracuda | Minority | | **Ocella** | `/OcellaWeb/planningSearch` | `Ocella` | **E** | None enforced (passive fingerprint at Arun) | Niche | | **StatMap horizoNext** ("Mirage") | `*-publicportal.statmap.co.uk`; `/horizonext` SPA; JSON API `/horizoNext/api/` | `Custom` | **F** | None — open JSON API | Small (ex-SwiftLG) | | **Agile Applications** | `planning.agileapplications.co.uk//`; "Citizen Portal Planning"; `*.sunagile.com` CSP | `Agile` | **G** | None — open JSON API (3 headers) | Small, growing (ex-SwiftLG) | | **TerraQuest PP2** | Next.js + `/__ENV.js` w/ `*.tqinfra.co.uk` API. NI: `planningregister.planningsystemni.gov.uk` | (indexed) | **I** | None — open JSON API (1 header) | All 11 NI authorities | | **Arcus** (Salesforce) | `*.my.site.com`/`*.force.com`/council CNAME; Lightning SPA; `Server: sfdcedge` | often mislabelled | — browser-only* | Guest Aura curl-able; blocked on unknown apex sigs | Small, growing | | **DEF Atrium** | `/Search/Results` POST + `__RequestVerificationToken`; `/Planning/Display?applicationNumber=`; `/Document/Download?module=PLA&…`; `/Content/def/` CSS | `Atrium` or `Custom` | **A** | None; Somerset adds a disclaimer-cookie gate | Small (incl. county registers) | | **Tascomi RSH** (Idox group) | `index.html?fa=` dispatcher; "Regulatory Services Hub" title; `AWSCaptcha.js` | `Tascomi` | — browser-only | **Enforcing AWS WAF challenge** (202 + `x-amzn-waf-action`) | Small, growing (ex-PE) | | **Idox Publisher** (docs host) | `d0cs.*` host; `/Publisher/mvc/listDocuments?identifier=…&ref=…`; `/publisher/idoxui/` CSS | `Custom` | **J** | None seen (downloads session-gated) | Docs module only — pairs with bespoke registers (Colchester) | | **Custom / bespoke** | none of the above | `Custom` | treat A as a template | Varies | Long tail | \* Arcus public registers have no anonymous API surface — treat as browser-only and hand the user a deep link. See the Arcus section. Detection cheat-sheet for an unknown portal: `.do` actions + `keyVal` = Idox (any base path); `/swiftlg/apas/run/` = SwiftLG; `/Northgate/PlanningExplorer/` = Northgate PE; `/NECSWS/ES/Presentation/` = NEC Assure; `/OcellaWeb/` = Ocella; `*-publicportal.statmap.co.uk` = StatMap; `planning.agileapplications.co.uk/` / "Citizen Portal Planning" = Agile; Next.js + `/__ENV.js` on `*.tqinfra.co.uk` = TerraQuest PP2; Salesforce `*.force.com`/`my.site.com`/`Server: sfdcedge` = Arcus; `civica.loader.js` = Civica; `/Search/Results` + `__RequestVerificationToken` + `/Document/Download?module=PLA` = DEF Atrium; `index.html?fa=` dispatcher + "Regulatory Services Hub" = Tascomi (browser-only); `/CMWebDrawer/` = HP TRIM docs host (append `&format=json`); `/Publisher/mvc/listDocuments` + `/publisher/idoxui/` assets = Idox Publisher docs host (Recipe J — a documents module paired with a bespoke register); `__VIEWSTATE` with none of the above = bespoke WebForms. **When the portal is a JS/SPA shell, fetch its runtime-config file** (`/__ENV.js`, `config.js`, or the app bundle) — for the open-API vendors (StatMap, Agile, TerraQuest) that file hands you the real API host, and often a tenant id/header you'll need. > **⚠ SwiftLG is being retired fast.** Of four historic SwiftLG councils tested in > Aug 2026, only Warwickshire still ran it — Mole Valley → StatMap, and Dudley, > Snowdonia, Pembrokeshire → Agile (old hosts dead or DNS gone). **Presume any > historic SwiftLG council has migrated until PlanIt's `planning_url` proves > otherwise.** The successors are open JSON APIs (Recipes F/G), so migration usually > makes retrieval *easier*, not harder. ## General procedure **Step 0 — get the portal URL + vendor.** Everything the recipes need is the *portal URL* and *vendor* for the user's council; the *reference* comes from the user. There are two paths, and the first one needs no external service at all. ### Fast path — known council: you don't need PlanIt **If the user gives you a planning reference and a council, and that council is in [`planning-portal-registry.json`](planning-portal-registry.json) with `status: tested-ok`, you are done resolving.** The registry row already gives you the portal URL, the vendor, and the per-council quirks. Go straight to the vendor's recipe and run it from the user's reference — search the reference → detail/keyVal → documents → download. **PlanIt is not in the loop for this case.** Reference + council + this skill (registry + recipe) is sufficient to retrieve the documents; do not call PlanIt just out of habit. (PlanIt's only remaining offer here would be the `n_documents` completeness figure — and the recipes get that from the portal's own documents page instead; see the completeness cross-check in the checklist. Skip PlanIt entirely on the fast path.) ### Resolution path — unknown/novel council: use PlanIt (or a council-site scrape) If the council is **not** in the registry (or its recorded portal looks stale — councils migrate), you need to resolve council → portal URL + vendor. Use the **PlanIt API** — the live national LPA → portal directory — instead of hand-hunting the council website: ```bash # Council -> portal base URL + vendor family (planning_url, scraper_type): curl -s "https://www.planit.org.uk/api/areas/json?area_type=planning&auths=" \ -A "planning-doc-search (contact: you@example.com)" # Reference -> the council application URL (the `url` field is the portal deep link): curl -s "https://www.planit.org.uk/api/applics/json?id_match=" \ -A "planning-doc-search (contact: you@example.com)" ``` ```bash # Need a fresh/recent application for an authority (e.g. to test with)? — curl -s "https://www.planit.org.uk/api/applics/json?auth=&pg_sz=3&recent=60" \ -A "planning-doc-search (contact: you@example.com)" ``` PlanIt rules: **always send an identifying User-Agent** (403 without one) and **back off on 429** (honour `Retry-After`). PlanIt gives you the council application URL + vendor but **not** document file links — follow the URL into the portal and apply the vendor recipe. (`planning.data.gov.uk` is spatial/policy constraints only — Article 4, conservation areas, listed buildings, local plans — **not** a document source.) Two PlanIt caveats: the **areas record can mislead** — Camden's `planning_url` points at a Socrata open-data dataset, not the portal; the *applics* records' `other_fields` (`docs_url`, `url`, `comment_url`) are the real portal pointers, so when the areas record looks odd, pull a recent applic and trust its URLs. And **merged authorities fragment**: post-2023 Somerset still files current district applications on the legacy district portals (Mendip/South Somerset Idox) while running a consolidated DEF Atrium register for county matters — PlanIt lists legacy districts as separate areas, and the per-application `url`/`docs_url` is the router that tells you which portal a given reference lives on. PlanIt shortcuts: applics records often carry **`other_fields.docs_url`** — the fully-formed Idox documents-tab URL *including the keyVal* — plus `n_documents` (a free completeness check against your scraped link count). The `url` field frequently embeds the portal's opaque record id too (Idox `keyVal=`, Ocella `planningDetails?reference=`, StatMap trailing internal id). When present you can skip a recipe's search steps and jump straight to the documents step — but for session-gated vendors (Idox) still hit the search page once first to get a session cookie. Derive a portal's base path from the areas `planning_url` by stripping the trailing `/search.do?...` page — do **not** assume `/online-applications/`. If PlanIt lacks the council, fall back to the council website: ```bash curl -s -L "https://www..gov.uk/view-and-track-planning-applications" -A "Mozilla/5.0" \ | grep -oiE 'href="http[^"]*"' | grep -iE 'planning|search|idox|publicaccess|civica|swiftlg|ocella' | sort -u ``` **PlanIt is a convenience, not a requirement.** All the vendor recipes need is the **portal URL + vendor**; the reference itself comes from the user. What PlanIt actually buys you is (a) resolving council → portal URL + vendor without hand-hunting the council site, and (b) shortcuts — `docs_url`/`keyVal`/internal ids that skip a recipe's search step, and `n_documents` as a free completeness check. If you already know a council's portal and vendor (e.g. it is in the registry with `status: tested-ok`), you can run the recipe directly from the user's reference and **use PlanIt only for the `n_documents` cross-check** — or skip it entirely. The registry is the offline substitute for (a); the recipe's own search step is the substitute for (b). **Steps 1–5 — identify the vendor** (table above) → **search** the reference → **extract the detail-page / keyVal** → **enumerate document links** → **download** each with the same session. Then **deliver** the files and **record the result in the registry** (`status`, `last_tested`, specializations). --- ## Recipe A — DEF Software "Atrium" Originally documented as "custom ASP.NET" from Welwyn Hatfield; **identified as the DEF Software Atrium product** — Somerset's consolidated register shares every form and endpoint (detect: `/Search/Results` POST, `__RequestVerificationToken`, `/Planning/Display?applicationNumber=`, `/Document/Download?module=PLA&recordNumber=…&planId=…`, `/Content/def/` CSS, `def.co.uk` privacy link; PlanIt `scraper_type: Atrium` or `Custom`). The search form is a server-rendered POST protected by an ASP.NET anti-forgery scheme: you need **two matching tokens** — the `__RequestVerificationToken` cookie *and* the hidden form-field token from the page HTML. They are validated as a pair, so steps 1–2 must run against the same cookie jar. With an exact application number, the search redirects straight to the detail page (Welwyn) or links to `/Planning/Display?applicationNumber=` (Somerset — the human reference is the key; no opaque internal id needed). **Somerset variant quirks**: a **disclaimer gate** precedes everything — POST `/Disclaimer/Accept?returnUrl=` with an **explicit `Content-Length: 0` header** (the front-end 411s a plain empty POST) → sets an `AcceptedDisclaimer` cookie (~1h expiry) that document downloads require (a cold download 302s back to the disclaimer). Search-scope flags differ (`SearchPlanning`/`SearchAppeals` + `AdvancedSearch=True`), and `DateReceivedFrom/To` (`dd/mm/yyyy`) date sweeps work for discovering references. ```bash # 1. Load homepage: capture session cookie (-c) and the hidden form token TOKEN=$(curl -s -c whc.txt "https://planning.welhat.gov.uk/" -A "Mozilla/5.0" \ | grep -o '__RequestVerificationToken" type="hidden" value="[^"]*' \ | sed 's/.*value="//') # 2. POST the search — token + app number + the four search-scope flags curl -s -b whc.txt -c whc.txt -A "Mozilla/5.0" -L \ "https://planning.welhat.gov.uk/Search/Results" \ --data-urlencode "__RequestVerificationToken=$TOKEN" \ --data-urlencode "SearchPlanning=True" \ --data-urlencode "SearchAppeals=True" \ --data-urlencode "SearchEnforcement=True" \ --data-urlencode "SearchTreePreservationOrders=True" \ --data-urlencode "ApplicationNumber=6/2026/1249/HOUSE" \ --data-urlencode "Address=" \ -o results.html # 3. Extract document download links (un-escape &) grep -oE 'href="/Document/Download[^"]*"' results.html \ | sed 's/href="//;s/"$//;s/&/\&/g' | sort -u # 4. Download each doc, reusing the session cookie. Links differ only in # planId / imageId / isPlan / fileName; recordNumber is the internal app id. curl -s -b whc.txt -A "Mozilla/5.0" -o "ApplicationFormRedacted.pdf" \ "https://planning.welhat.gov.uk/Document/Download?module=PLA&recordNumber=111526&planId=2173461&imageId=2&isPlan=False&fileName=ApplicationFormRedacted.pdf" ``` Notes: - Always send a real browser `User-Agent` (`-A "Mozilla/5.0 …"`). Some council portals reject curl's default UA. - The detail page also carries each document's description and created date next to the link — grep those out to label the files for the user. - Verify downloads are real PDFs (`file *.pdf`) — a bot block or session timeout returns an HTML error page with a `.pdf` name. --- ## Recipe B — Civica Portal360 (browser-only only where an enforcing Barracuda is present) **Barracuda is a per-council property, not a Civica one.** Of the four CivicaJson sites in PlanIt, Ashfield has no WAF, Waverley has passive Imperva Incapsula, and only St Albans runs *enforcing* Barracuda. So on most Civica sites the JSON API is fully curl-able — the browser fallback below is needed only when you actually get a `Blocked` page. ### B (curl) — the JSON API chain Everything is a JSON POST to the Civica `Handler.ashx` API. Two things are per-site; **read them from the search page's inline config**, don't hardcode: - `Civica.APIUrl=` → the API base path (`/civica/Resource/Civica/Handler.ashx/` at Ashfield; `/w2webparts/Resource/Civica/Handler.ashx/` at Waverley/St Albans; an absolute cross-host URL at Eastbourne). - `Civica.PortalSettings.PlanningApplicationRefType` → the `refType` (`GFPlanning` at Ashfield/Waverley; the old `PBDC` was St-Albans-specific). ```bash UA="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36" API="https://planning.ashfield.gov.uk/civica/Resource/Civica/Handler.ashx" # from Civica.APIUrl RT="GFPlanning" # from PlanningApplicationRefType REF="V/2026/0515" # 1. Search (JSON POST — NOT the old keyText GET). -> KeyObjects[].KeyNumber curl -s -b civ.txt -c civ.txt -A "$UA" -H "Content-Type: application/json" \ "$API/keyobject/pagedsearch" \ -d "{\"refType\":\"$RT\",\"fromRow\":1,\"toRow\":10,\"searchFields\":{\"FullTextSearch\":\"$REF\"}}" # 2. Document list for that KeyNumber (KeyText is the constant "Subject") curl -s -b civ.txt -c civ.txt -A "$UA" -H "Content-Type: application/json" \ "$API/doc/list" \ -d "{\"KeyNumb\":244982,\"KeyText\":\"Subject\",\"RefType\":\"$RT\",\"ProcessNo\":\"\",\"PageSize\":50}" # 3. Download one document by DocNo -> application/pdf curl -s -b civ.txt -A "$UA" -o "SiteLocationPlan.pdf" \ "$API/doc/pagestream?DocNo=19338083&pdf=true&filename=SiteLocationPlan.pdf" ``` Notes: a `500` with a JSON error body (LogRef) = wrong params, **not** a block — distinct from a Barracuda `Blocked` HTML page. Empty `searchFields` returns zero rows by design. Doc endpoints are defined in `civica.documents.js` under `presentation/bundle/` (not `civica.common.js`). PlanIt's applics `url` carries `?RefType=&KeyNo=` for Civica sites — skip step 1. More Civica facts: - **Zip-all** — `GET doc/list/zipstream?KeyNumb=&KeyText=&RefType=&ProcessNo=` returns one zip of the entire document set. Prefer it for whole-application pulls; verify `PK` magic + entry count. - **Two keying schemes**: number-keyed (Ashfield/Waverley — `KeyNumb=` from pagedsearch, `KeyText:"Subject"`) vs **reference-keyed** (Great Yarmouth — `KeyNumb:0`, `KeyText:""`, `RefType:"PLANNINGCASE"`). The viewer deep-link fragment tells you which: `#VIEW?…&KeyNo=` vs `…&KeyText=`. - **Silent search trap**: `FullTextSearch` can be ignored on some installs (Great Yarmouth returned the whole 30,992-row register) — always check `TotalRows` == expected; the honoured field there was `searchFields:{"KeyNo":""}`. - `doc/list` rows may lack `FileName` (Waverley) — label from `DocDesc` + `FileExtension`; the `filename=` param on pagestream is client-chosen anyway. - pagestream Content-Type can be `application/PDF` (uppercase) — match case-insensitively. Native `.docx` sources are converted to real PDF by `pdf=true`. - Non-standard **ports** occur (Waverley `:4443`) — take host:port verbatim from PlanIt's `planning_url`. ### B (browser) — only when Barracuda actually enforces (St Albans) `planningapplications.stalbans.gov.uk`. Here the page shell renders but every XHR/API and document call returns a `Blocked` page. The server sends a ~67 KB obfuscated JavaScript fingerprinting challenge (loaded via a `