Planning Document Search

Planning Document Search Skill

Download the documents attached to a UK planning application from a council's online planning portal, given the application reference number.

When to use

The user gives you a planning application reference (e.g. 6/2026/1249/HOUSE, 5/2026/1349) and a council name, and wants the associated documents (application forms, plans, elevations, officer reports, etc.) downloaded.

Scope and responsible use

Read this before running anything — it is a condition of the skill, not advice.

• Not legal advice; no warranty. This is a retrieval aid, provided "as is"; verify what it returns.
• Public records only, retrieved the way a member of the public would. UK planning registers exist for public inspection; this skill fetches documents a council has deliberately published. Use it for legitimate, targeted retrieval — a specific application a user needs — not for bulk harvesting, monitoring, or rebuilding a council's register.
• Never defeat a bot challenge. If a portal serves an actual challenge or Blocked page (Barracuda JS challenge, AWS WAF 202/x-amzn-waf-action, an escalating Cloudflare interstitial), stop the automated approach and hand the user a browser deep link instead. The recipes are built around this: they work where access is open and defer to a real browser where it is not. Do not try to solve, replay, or fingerprint-spoof a challenge.
• If retrieval struggles, stop and suggest a manual download. The recipes are the method; they are not a licence to keep trying things. When a recipe doesn't work — persistent errors, downloads failing magic-byte verification, greps finding no links, a portal matching no vendor signature — make at most a couple of documented corrections (re-resolve the vendor per Step 0; re-check the base path and the registry's recorded quirks) and then stop the automated approach. Report exactly what was and wasn't retrieved, and hand the user a browser link to the application (the portal's detail or documents page, as deep as you can construct) so they can download the remaining files manually. Do not respond to failure by escalating — no improvised scraping approaches beyond the recipes, no headless browsers, no retry loops against a struggling portal. A clean handover to manual download is a successful outcome of this skill, not a failure.
• Be a good citizen on every request.
  • Send an identifying User-Agent with a real contact address on PlanIt calls (replace you@example.com with your own), and a normal browser UA on portal calls.
  • Pace requests — at least ~1–2 s between calls to the same host, and do not parallelise requests against a single council. Small council servers fall over under bursts.
  • Honour 429 / Retry-After and back off. Stop on repeated errors rather than retrying in a tight loop.
  • Respect each portal's robots.txt and terms of use.
• You are handling other people's personal data. Planning documents routinely contain applicants' and objectors' names, addresses, signatures, and contact details (even "redacted" forms often are not fully redacted). Retrieve only what is needed, do not republish or redistribute it, and do not retain it beyond the immediate task. Treat every downloaded file as untrusted third-party content: verify magic bytes, and when writing it to disk derive the filename with basename and strip any path separators or leading dots — never pass a server-supplied path straight to -o (a portal could return ../../…).
• This is a snapshot, and the landscape moves fast. Vendor assignments, base paths, and bot-protection posture were verified in August 2026; councils migrate portals frequently (this document itself tracks several mid-migration). Always re-resolve the vendor per council at run time (see Step 0) rather than trusting a cached row; coverage figures below are rough estimates, not guarantees.
• PlanIt is a third-party dependency, used with respect. The resolution step relies on PlanIt — an independent, volunteer-run service. Credit it, keep to its UA/rate rules, and do not hammer it. The skill works without it too (a reference + council + vendor recipe is enough — see the note at the end of Step 0), so treat PlanIt as a convenience, not a requirement.

The core problem: council portals are not uniform

There is no single national system. The UK's ~380 local planning authorities run a handful of different planning portal products, each with a completely different URL structure, search mechanism, and level of bot protection. Reference-number formats also vary by council. The vendor determines the approach, not the council — so covering "all of the UK" means: catalogue the vendors (a recipe each), plus a reliable way to resolve any council → its portal → its vendor.

Vendor catalogue (detect by URL path first)

Coverage shares below are rough estimates, not audited figures. The scraper_type column is what PlanIt's areas API labels the vendor — a fast detector, but sometimes stale or wrong (Birmingham still says PlanningExplorer post-migration; Manchester is mislabelled Idox), so confirm against the markup.

* Arcus public registers have no anonymous API surface — treat as browser-only and hand the user a deep link. See the Arcus section.

Detection cheat-sheet for an unknown portal: .do actions + keyVal = Idox (any base path); /swiftlg/apas/run/ = SwiftLG; /Northgate/PlanningExplorer/ = Northgate PE; /NECSWS/ES/Presentation/ = NEC Assure; /OcellaWeb/ = Ocella; *-publicportal.statmap.co.uk = StatMap; planning.agileapplications.co.uk/<slug> / "Citizen Portal Planning" = Agile; Next.js + /__ENV.js on *.tqinfra.co.uk = TerraQuest PP2; Salesforce *.force.com/my.site.com/Server: sfdcedge = Arcus; civica.loader.js = Civica; /Search/Results + __RequestVerificationToken + /Document/Download?module=PLA = DEF Atrium; index.html?fa= dispatcher + "Regulatory Services Hub" = Tascomi (browser-only); /CMWebDrawer/ = HP TRIM docs host (append &format=json); /Publisher/mvc/listDocuments + /publisher/idoxui/ assets = Idox Publisher docs host (Recipe J — a documents module paired with a bespoke register); __VIEWSTATE with none of the above = bespoke WebForms. When the portal is a JS/SPA shell, fetch its runtime-config file (/__ENV.js, config.js, or the app bundle) — for the open-API vendors (StatMap, Agile, TerraQuest) that file hands you the real API host, and often a tenant id/header you'll need.

⚠ SwiftLG is being retired fast. Of four historic SwiftLG councils tested in Aug 2026, only Warwickshire still ran it — Mole Valley → StatMap, and Dudley, Snowdonia, Pembrokeshire → Agile (old hosts dead or DNS gone). Presume any historic SwiftLG council has migrated until PlanIt's planning_url proves otherwise. The successors are open JSON APIs (Recipes F/G), so migration usually makes retrieval easier, not harder.

General procedure

Step 0 — get the portal URL + vendor. Everything the recipes need is the portal URL and vendor for the user's council; the reference comes from the user. There are two paths, and the first one needs no external service at all.

Fast path — known council: you don't need PlanIt

If the user gives you a planning reference and a council, and that council is in planning-portal-registry.json with status: tested-ok, you are done resolving. The registry row already gives you the portal URL, the vendor, and the per-council quirks. Go straight to the vendor's recipe and run it from the user's reference — search the reference → detail/keyVal → documents → download. PlanIt is not in the loop for this case. Reference + council + this skill (registry + recipe) is sufficient to retrieve the documents; do not call PlanIt just out of habit.

(PlanIt's only remaining offer here would be the n_documents completeness figure — and the recipes get that from the portal's own documents page instead; see the completeness cross-check in the checklist. Skip PlanIt entirely on the fast path.)

Resolution path — unknown/novel council: use PlanIt (or a council-site scrape)

If the council is not in the registry (or its recorded portal looks stale — councils migrate), you need to resolve council → portal URL + vendor. Use the PlanIt API — the live national LPA → portal directory — instead of hand-hunting the council website:

# Council -> portal base URL + vendor family (planning_url, scraper_type):
curl -s "https://www.planit.org.uk/api/areas/json?area_type=planning&auths=<council>" \
  -A "planning-doc-search (contact: you@example.com)"

# Reference -> the council application URL (the `url` field is the portal deep link):
curl -s "https://www.planit.org.uk/api/applics/json?id_match=<ref>" \
  -A "planning-doc-search (contact: you@example.com)"

# Need a fresh/recent application for an authority (e.g. to test with)? —
curl -s "https://www.planit.org.uk/api/applics/json?auth=<council>&pg_sz=3&recent=60" \
  -A "planning-doc-search (contact: you@example.com)"

PlanIt rules: always send an identifying User-Agent (403 without one) and back off on 429 (honour Retry-After). PlanIt gives you the council application URL + vendor but not document file links — follow the URL into the portal and apply the vendor recipe. (planning.data.gov.uk is spatial/policy constraints only — Article 4, conservation areas, listed buildings, local plans — not a document source.)

Two PlanIt caveats: the areas record can mislead — Camden's planning_url points at a Socrata open-data dataset, not the portal; the applics records' other_fields (docs_url, url, comment_url) are the real portal pointers, so when the areas record looks odd, pull a recent applic and trust its URLs. And merged authorities fragment: post-2023 Somerset still files current district applications on the legacy district portals (Mendip/South Somerset Idox) while running a consolidated DEF Atrium register for county matters — PlanIt lists legacy districts as separate areas, and the per-application url/docs_url is the router that tells you which portal a given reference lives on.

PlanIt shortcuts: applics records often carry other_fields.docs_url — the fully-formed Idox documents-tab URL including the keyVal — plus n_documents (a free completeness check against your scraped link count). The url field frequently embeds the portal's opaque record id too (Idox keyVal=, Ocella planningDetails?reference=, StatMap trailing internal id). When present you can skip a recipe's search steps and jump straight to the documents step — but for session-gated vendors (Idox) still hit the search page once first to get a session cookie. Derive a portal's base path from the areas planning_url by stripping the trailing /search.do?... page — do not assume /online-applications/.

If PlanIt lacks the council, fall back to the council website:

curl -s -L "https://www.<council>.gov.uk/view-and-track-planning-applications" -A "Mozilla/5.0" \
  | grep -oiE 'href="http[^"]*"' | grep -iE 'planning|search|idox|publicaccess|civica|swiftlg|ocella' | sort -u

PlanIt is a convenience, not a requirement. All the vendor recipes need is the portal URL + vendor; the reference itself comes from the user. What PlanIt actually buys you is (a) resolving council → portal URL + vendor without hand-hunting the council site, and (b) shortcuts — docs_url/keyVal/internal ids that skip a recipe's search step, and n_documents as a free completeness check. If you already know a council's portal and vendor (e.g. it is in the registry with status: tested-ok), you can run the recipe directly from the user's reference and use PlanIt only for the n_documents cross-check — or skip it entirely. The registry is the offline substitute for (a); the recipe's own search step is the substitute for (b).

Steps 1–5 — identify the vendor (table above) → search the reference → extract the detail-page / keyVal → enumerate document links → download each with the same session. Then deliver the files and record the result in the registry (status, last_tested, specializations).

Recipe A — DEF Software "Atrium"

Originally documented as "custom ASP.NET" from Welwyn Hatfield; identified as the DEF Software Atrium product — Somerset's consolidated register shares every form and endpoint (detect: /Search/Results POST, __RequestVerificationToken, /Planning/Display?applicationNumber=, /Document/Download?module=PLA&recordNumber=…&planId=…, /Content/def/ CSS, def.co.uk privacy link; PlanIt scraper_type: Atrium or Custom).

The search form is a server-rendered POST protected by an ASP.NET anti-forgery scheme: you need two matching tokens — the __RequestVerificationToken cookie and the hidden form-field token from the page HTML. They are validated as a pair, so steps 1–2 must run against the same cookie jar.

With an exact application number, the search redirects straight to the detail page (Welwyn) or links to /Planning/Display?applicationNumber=<enc-ref> (Somerset — the human reference is the key; no opaque internal id needed).

Somerset variant quirks: a disclaimer gate precedes everything — POST /Disclaimer/Accept?returnUrl=<path> with an explicit Content-Length: 0 header (the front-end 411s a plain empty POST) → sets an AcceptedDisclaimer cookie (~1h expiry) that document downloads require (a cold download 302s back to the disclaimer). Search-scope flags differ (SearchPlanning/SearchAppeals + AdvancedSearch=True), and DateReceivedFrom/To (dd/mm/yyyy) date sweeps work for discovering references.

# 1. Load homepage: capture session cookie (-c) and the hidden form token
TOKEN=$(curl -s -c whc.txt "https://planning.welhat.gov.uk/" -A "Mozilla/5.0" \
  | grep -o '__RequestVerificationToken" type="hidden" value="[^"]*' \
  | sed 's/.*value="//')

# 2. POST the search — token + app number + the four search-scope flags
curl -s -b whc.txt -c whc.txt -A "Mozilla/5.0" -L \
  "https://planning.welhat.gov.uk/Search/Results" \
  --data-urlencode "__RequestVerificationToken=$TOKEN" \
  --data-urlencode "SearchPlanning=True" \
  --data-urlencode "SearchAppeals=True" \
  --data-urlencode "SearchEnforcement=True" \
  --data-urlencode "SearchTreePreservationOrders=True" \
  --data-urlencode "ApplicationNumber=6/2026/1249/HOUSE" \
  --data-urlencode "Address=" \
  -o results.html

# 3. Extract document download links (un-escape &)
grep -oE 'href="/Document/Download[^"]*"' results.html \
  | sed 's/href="//;s/"$//;s/&/\&/g' | sort -u

# 4. Download each doc, reusing the session cookie. Links differ only in
#    planId / imageId / isPlan / fileName; recordNumber is the internal app id.
curl -s -b whc.txt -A "Mozilla/5.0" -o "ApplicationFormRedacted.pdf" \
  "https://planning.welhat.gov.uk/Document/Download?module=PLA&recordNumber=111526&planId=2173461&imageId=2&isPlan=False&fileName=ApplicationFormRedacted.pdf"

Notes:

• Always send a real browser User-Agent (-A "Mozilla/5.0 …"). Some council portals reject curl's default UA.
• The detail page also carries each document's description and created date next to the link — grep those out to label the files for the user.
• Verify downloads are real PDFs (file *.pdf) — a bot block or session timeout returns an HTML error page with a .pdf name.

Recipe B — Civica Portal360 (browser-only only where an enforcing Barracuda is present)

Barracuda is a per-council property, not a Civica one. Of the four CivicaJson sites in PlanIt, Ashfield has no WAF, Waverley has passive Imperva Incapsula, and only St Albans runs enforcing Barracuda. So on most Civica sites the JSON API is fully curl-able — the browser fallback below is needed only when you actually get a Blocked page.

B (curl) — the JSON API chain

Everything is a JSON POST to the Civica Handler.ashx API. Two things are per-site; read them from the search page's inline config, don't hardcode:

• Civica.APIUrl= → the API base path (/civica/Resource/Civica/Handler.ashx/ at Ashfield; /w2webparts/Resource/Civica/Handler.ashx/ at Waverley/St Albans; an absolute cross-host URL at Eastbourne).
• Civica.PortalSettings.PlanningApplicationRefType → the refType (GFPlanning at Ashfield/Waverley; the old PBDC was St-Albans-specific).

UA="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36"
API="https://planning.ashfield.gov.uk/civica/Resource/Civica/Handler.ashx"   # from Civica.APIUrl
RT="GFPlanning"                                                              # from PlanningApplicationRefType
REF="V/2026/0515"

# 1. Search (JSON POST — NOT the old keyText GET). -> KeyObjects[].KeyNumber
curl -s -b civ.txt -c civ.txt -A "$UA" -H "Content-Type: application/json" \
  "$API/keyobject/pagedsearch" \
  -d "{\"refType\":\"$RT\",\"fromRow\":1,\"toRow\":10,\"searchFields\":{\"FullTextSearch\":\"$REF\"}}"

# 2. Document list for that KeyNumber (KeyText is the constant "Subject")
curl -s -b civ.txt -c civ.txt -A "$UA" -H "Content-Type: application/json" \
  "$API/doc/list" \
  -d "{\"KeyNumb\":244982,\"KeyText\":\"Subject\",\"RefType\":\"$RT\",\"ProcessNo\":\"\",\"PageSize\":50}"

# 3. Download one document by DocNo -> application/pdf
curl -s -b civ.txt -A "$UA" -o "SiteLocationPlan.pdf" \
  "$API/doc/pagestream?DocNo=19338083&pdf=true&filename=SiteLocationPlan.pdf"

Notes: a 500 with a JSON error body (LogRef) = wrong params, not a block — distinct from a Barracuda Blocked HTML page. Empty searchFields returns zero rows by design. Doc endpoints are defined in civica.documents.js under <base>presentation/bundle/ (not civica.common.js). PlanIt's applics url carries ?RefType=&KeyNo= for Civica sites — skip step 1.

More Civica facts:

• Zip-all — GET <base>doc/list/zipstream?KeyNumb=<n>&KeyText=<kt>&RefType=<rt>&ProcessNo= returns one zip of the entire document set. Prefer it for whole-application pulls; verify PK magic + entry count.
• Two keying schemes: number-keyed (Ashfield/Waverley — KeyNumb=<int> from pagedsearch, KeyText:"Subject") vs reference-keyed (Great Yarmouth — KeyNumb:0, KeyText:"<planning ref>", RefType:"PLANNINGCASE"). The viewer deep-link fragment tells you which: #VIEW?…&KeyNo=<n> vs …&KeyText=<ref>.
• Silent search trap: FullTextSearch can be ignored on some installs (Great Yarmouth returned the whole 30,992-row register) — always check TotalRows == expected; the honoured field there was searchFields:{"KeyNo":"<ref>"}.
• doc/list rows may lack FileName (Waverley) — label from DocDesc + FileExtension; the filename= param on pagestream is client-chosen anyway.
• pagestream Content-Type can be application/PDF (uppercase) — match case-insensitively. Native .docx sources are converted to real PDF by pdf=true.
• Non-standard ports occur (Waverley :4443) — take host:port verbatim from PlanIt's planning_url.

B (browser) — only when Barracuda actually enforces (St Albans)

planningapplications.stalbans.gov.uk. Here the page shell renders but every XHR/API and document call returns a Blocked page. The server sends a ~67 KB obfuscated JavaScript fingerprinting challenge (loaded via a <script src="/bnith__…"> tag) that a real browser must execute: it computes a device fingerprint, posts it back, and only then is a clearance cookie issued that unlocks the API. Consequences:

• The top-level HTML page loads for curl (Barracuda allows the document navigation so it can serve the challenge), but every XHR/API and document call returns a Blocked page. No clearance cookie is ever set for a non-JS client.
• Reusing cookies from the HTML fetch does not help — there is no valid cookie to reuse until the JS challenge is solved.
• A headless browser (Playwright/Puppeteer) might work, but Barracuda is specifically designed to detect and block headless Chromium, so success is unlikely and not worth a heavy install as a first move.

How to get past it: use a real browser session

1. Claude in Chrome extension (preferred if available) — drives the user's real Chrome, which passes the challenge invisibly because it's a genuine browser session. Requires the extension installed and signed in to the same account. (In this session it was "not connected", which blocked the automated route.)

2. In-app Browser pane — was blocked by environment policy for this domain, so unavailable here. Worth trying for other councils.

3. Hand the user a one-click deep link — the most reliable fallback. The full-text search deep link opens the application directly in the user's own browser:

   https://planningapplications.stalbans.gov.uk/planning/search-applications?civica.query.FullTextSearch=5%2F2026%2F1349
   

   (URL-encode the reference: / → %2F.) Civica deep-links to a specific record use the fragment form #VIEW?RefType=PBDC&KeyNo=<internal-id>, where KeyNo is the internal record id exposed in search-result links.

Useful Civica internals (for when you do have a browser session)

• Config globals in the page: Civica.APIUrl="/w2webparts/Resource/Civica/Handler.ashx/", Civica.DocumentViewerUrl="/my-requests/document-viewer".
• Search service: API.KeyObject → keyobject/search and keyobject/pagedsearch.
• The planning search/view widget is keyobjectsearchandview with RefType=PBDC; its display fields include ref_no, application_address, app_status, decision_date, etc.
• JS bundles live under /civica/Bundles/ (civica.common.js holds the API service definitions) — useful for discovering endpoint names without a browser.

Recipe C — Idox Public Access

The highest-value recipe — Idox powers the majority of UK LPAs. The base path varies — /online-applications/ (common), /idoxpa-web/ (Edinburgh), /publicaccess/ (Cardiff) all seen, and the wrong one 403s — so treat it as a per-council variable and detect Idox by its endpoints (search.do, keyVal, /files/{hex}/pdf/). Note Idox increasingly lives on vendor-cloud hostnames (*.idoxcloud, Cardiff's *.wales) rather than council domains, so resolve via PlanIt rather than guessing <council>.gov.uk. It is session-gated (a JSESSIONID from the search page is required; results endpoints 500 without it — note JSESSIONID is HttpOnly: rely on curl's cookie jar, not a Set-Cookie grep) and the search form carries a _csrf token you must echo back (format varies: opaque hex or lowercase UUID). Validated end-to-end against nine councils in August 2026, including Scottish and Welsh authorities.

UA="Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0 Safari/537.36"
BASE="https://planningaccess.york.gov.uk/online-applications"   # <- swap per council; base path VARIES (e.g. Edinburgh /idoxpa-web)
BP=$(echo "$BASE" | sed -E 's#https?://[^/]+##')                # path part, e.g. /online-applications
REF="24/00593/FUL"

# 1. Establish session; capture JSESSIONID (-c) and the _csrf hidden token
curl -s -c idox.txt -A "$UA" "$BASE/search.do?action=simple&searchType=Application" -o s.html
CSRF=$(grep -oE 'name="_csrf"[^>]*value="[^"]*"' s.html | grep -oE 'value="[^"]*"' | sed 's/value="//;s/"//')

# 2. POST the simple search. All four fields are required — WITHOUT
#    searchCriteria.simpleSearch=true the server returns "Too many results".
curl -s -b idox.txt -c idox.txt -A "$UA" -L "$BASE/simpleSearchResults.do?action=firstPage" \
  --data-urlencode "_csrf=$CSRF" \
  --data-urlencode "searchType=Application" \
  --data-urlencode "searchCriteria.simpleSearchString=$REF" \
  --data-urlencode "searchCriteria.simpleSearch=true" \
  -o results.html

# 3. Extract the keyVal (opaque ~13-char token, NOT the human reference).
#    An exact-ref search may 302 to the detail page OR return a one-row results
#    list (both observed) — the grep covers both.
KEYVAL=$(grep -oE 'keyVal=[A-Z0-9]+' results.html | head -1 | cut -d= -f2)

# 4. Documents tab -> scrape file hrefs. PDFs are $BP/files/{32-HEX}/pdf/{name}.pdf
#    but NON-PDF attachments (JPG photos/plans) OMIT the /pdf/ segment
#    ($BP/files/{32-HEX}/{name}.jpg — Leeds had 4 of 6 docs like this), so do
#    NOT anchor the grep on /pdf/. Also grep against $BP, NOT a hardcoded
#    /online-applications/ — either mistake silently yields missing files.
#    Reconcile your link count against the documents tab's own total (it lists
#    one row per document) — see the completeness cross-check in the checklist.
DOCS="$BASE/applicationDetails.do?activeTab=documents&keyVal=$KEYVAL"
curl -s -b idox.txt -c idox.txt -A "$UA" "$DOCS" -o docs.html
grep -oE "$BP/files/[A-Fa-f0-9]+/[^\"]+" docs.html | sort -u > files.txt

# 5. Download each file WITH a Referer header = the documents-tab URL.
#    Some councils (verified: RBWM, Greater Cambridge) 404 the file without it;
#    harmless on the councils that don't need it, so always send it.
HOST=$(echo "$BASE" | grep -oE 'https://[^/]+')
while read -r f; do
  name=$(basename "$f")
  curl -s -b idox.txt -A "$UA" -e "$DOCS" "$HOST$f" -o "$name"
done < files.txt

Idox gotchas:

• Base path varies — five variants observed: /online-applications/ (common), /idoxpa-web/ (Edinburgh), /publicaccess/ (Cardiff), /PlanningData-live/ (Stockport), /wam/ (Highland). Derive it from PlanIt's planning_url; never hardcode it in greps (see step 4).
• keyVal is opaque and per-application — scrape it from the results/detail link; you cannot construct it from the reference. (PlanIt often hands it to you in docs_url / url — see the PlanIt shortcuts above.)
• File-GET gating varies three ways — always send BOTH the session jar and the Referer: session-gated (Glasgow, Leeds, Stockport, Highland — a cold GET returns an HTML 404 under the .pdf name, Referer irrelevant), Referer-gated (RBWM, Greater Cambridge — 404 without -e <documents-tab URL>), or neither (York, Tendring, East Suffolk, Edinburgh). Consequence: never "optimise away" the search.do session step when jumping via PlanIt's docs_url.
• Non-PDF attachments lack the /pdf/ path segment — grep /files/{hex}/ without anchoring on /pdf/, verify magic bytes per file, and reconcile the link count against the documents tab's own row count (this is what catches silent misses). PlanIt n_documents, when you have it, is a secondary check only — it can lag (Highland 8 vs 9 real; Dudley 20 vs 22) so treat it as a lower bound, and note PlanIt omits docs_url entirely on applications it has seen zero documents for.
• Session ~30 min idle timeout; refresh (search.do) on long crawls.
• Pace requests ~1–2s — small council servers throw transient 000/500 on bursts.
• WAF minority — most Idox sites take plain curl; a few are behind Cloudflare (pass with the real browser UA above) and a rare few behind an enforcing Barracuda (browser-only, like Civica — see Recipe B). Barracuda cookies ≠ blocked: East Suffolk sets BNIS_/BNES_ cookies in passive mode and plain curl works fine — the actionable signal is an actual served JS challenge or Blocked page, not BN* cookies.
• recaptcha markup ≠ CAPTCHA enforced — Idox comment/copy-request widgets carry recaptcha classes; search and downloads are unaffected.
• The advanced address search needs caseAddressType as well as _csrf — posting searchCriteria.address to advancedSearchResults.do with a freshly-scraped, valid _csrf still returns "No results found" unless caseAddressType=Application rides along. This is the same silent-zero failure mode as a stale _csrf, from a different cause, so a zero on an address search is never evidence the site has no applications until you have re-run it with both fields. (Isolated at Cheltenham, alternating with/without on fresh cookie jars.) Page results with pagedSearchResults.do?action=page&searchCriteria.page=N.
• An application can be withdrawn from public view while its keyVal still resolves — the deep link returns HTTP 200, not a 404, carrying "This application is no longer available for viewing. It may have been removed or restricted from public viewing." The case then disappears from simple search, advanced search and the weekly/monthly lists together, so no portal route recovers it. Detect it on the string, not the status code, and report it as removed rather than as a retrieval failure — they are different findings for the user. A national aggregator (PlanIt) that scraped the case before removal keeps the reference, description, dates, keyVal and n_documents; that metadata is recoverable, the documents are not.
• The weekly/monthly lists are a second route onto a case — search.do?action=weeklyList takes a week value in the portal's own display format (e.g. 29 Jun 2026) plus optional parish/ward codes, and lists EIA screening and other non-application case types alongside applications. Useful as an independent check that a case really is absent rather than merely unmatched by your search terms.

Recipe D — Northgate (two distinct products)

D1 · SwiftLG APAS (/swiftlg/apas/run/) — ⚠ dying fast

Oracle-backed register, plain curl + cookie jar, no bot protection — but presume any historic SwiftLG council has migrated (3 of 4 tested had: Mole Valley → StatMap; Dudley, Snowdonia, Pembrokeshire → Agile). Confirm the portal still answers on /swiftlg/apas/run/ via PlanIt before applying this. Validated end-to-end at Warwickshire; the whole chain is stateless (works cold, no cookies/CSRF/Referer).

• Search: POST WPHAPPCRITERIA with APNID.MAINBODY.WPACIS.1=<ref> and SEARCHBUTTON.MAINBODY.WPACIS.1=<any non-empty value> — the SEARCHBUTTON param must be present (omitting it returns a 110-byte stub); other fields omittable. Results render inline in the POST response (WPHAPPSEARCHRES.displayResultsURL is only the re-display/pagination URL). To find refs without PlanIt (which can't resolve county councils): date-sweep with REGFROMDATE/REGTODATE.MAINBODY.WPACIS.1 (dd/mm/yyyy).
• Detail: GET WPHAPPDETAIL.DisplayUrl?theApnID=<REF> (theApnID is literally the reference, slashes unencoded). Do NOT tab-hunt with theTabNo — skinned installs render all tabs in one page and non-1 values throw WCHINTERROR; fetch with theTabNo=1 (or omit) and grep the whole page for document links.
• Documents: hrefs are <SKIN>DISPLAYMEDIA.showImage?theSeqNo=<n>&theApnkey=<key>&theModule=1 (the handler prefix is per-council skin — WCH at Warwickshire vs the generic WPH). This returns a 72-byte meta-refresh stub, which curl -L does not follow — parse URL=../MediaTemp/{apnkey}-{seqno}.pdf out of it, then GET that with -L (it 302s to /swiftlg/MediaTemp/…) to get the PDF.

D2 · Planning Explorer (/Northgate/PlanningExplorer/)

Classic ASP.NET WebForms — round-trip __VIEWSTATE, __VIEWSTATEGENERATOR and __EVENTVALIDATION (all three), same cookie jar (ASP.NET_SessionId).

• GET GeneralSearch.aspx, capture the three hidden tokens (VIEWSTATE can be ~96–115 KB — build the POST body with a script, not shell args).

• Reference lookup: POST the MINIMAL field set — the three tokens + txtApplicationNumber=<ref> + csbtnSearch=Search (rbGroup optional). Every extra empty field is WAF-bait: Runnymede's origin WAF 403s any POST containing an empty cboSelectDateValue. (The txtProposal + date-range route is for proposal/date sweeps, not reference lookups.)

• Search 302s to Generic/StdResults.aspx?...&PS=10&XMLLoc=/…/XMLtemp/<session>/<guid>.xml. Robust PARAM0 source: skip StdResults entirely — take XMLLoc from the 302 Location header and GET that XML directly (same cookie jar); <PK> in it is PARAM0. This works even where StdResults is WAF-blocked (Runnymede 403s it wholesale) and is cleaner than scraping results HTML. Caveat: the Location header can contain raw spaces — percent-encode it yourself; don't rely on curl -L.

• Detail page on skinned sites is Generic/StdDetails.aspx?…&TYPE=PL/PlanningPK.xml&PARAM0=<id>&XSLT=… (the opaque key is PARAM0, not keyVal). Keep slashes literal in the TYPE/XSLT/XMLSIDE params — required at Wandsworth (%2F → 500), harmless at Runnymede. Detail hrefs carry literal CRLF/tabs — strip whitespace.

• The documents module varies per authority — both councils examined replaced the standard Northgate one, differently, and both work cold (no prior session needed if you have the ref):

  • Wandsworth: bespoke cross-host IAM — planning2 host, comments.aspx?case=<ref>, WebForms category-expand postbacks, then IAM/IAMLink.aspx?docid=<n> → 302 → PDF.
  • Runnymede: NEC "Public Access" MVC on a docs.<council> host (a naming collision — not Idox) — …/PublicAccess_LIVE/SearchResult/RunThirdPartySearch?FileSystemId=PL&FOLDER1_REF=<ref> (constructible from the ref); the doc list is embedded in the HTML as var model = {…Rows:[{Guid,Doc_Type,Date_Received,Doc_Ref2}]}; download GET …/Document/ViewDocument?id=<Guid> (param must be id — check Content-Type).
  • Camden: HP TRIM / Content Manager "CMWebDrawer" on a camdocs.<council> host — fully constructible from the ref, stateless: GET /CMWebDrawer/PlanRec?q=recContainer:%22<enc-ref>%22&format=json (ServiceStack JSON: Results[].Uri, RecordTitle, TotalResults/HasMoreItems, &pageSize= honoured) then GET /CMWebDrawer/Record/<Uri>/file/document?inline → PDF. The HTML view double-lists each doc — dedupe on record id. The &format=json trick likely generalises to any TRIM WebDrawer council.

  PlanIt's docs_url points at whichever module a council uses — record it per council.

• WAF: Cloudflare has three modes on PE sites: absent (Wandsworth), passive (Runnymede — any UA passes; the old "empty/default UA gets 403" claim did not reproduce), and score/rate-based enforcing (Camden's register: the first ~2 curl requests reach origin, then everything — including previously-working URLs — gets the interactive challenge). On a score-based site, grab what you need in the first requests; "worked a minute ago, 403 now" = escalation, don't burn retries. Watch also for an origin WAF with narrow triggers (Runnymede's StdResults/ cboSelectDateValue blocks above).

• NEC's Redirection/redirect.aspx deep links (PlanIt url for some PE councils) redirect via a JavaScript stub, not a 302 — parse document.location.href out of the body; curl -L won't follow.

• NB migration: the PE fleet is dying but its successors are diverse — Birmingham → NEC Assure (Recipe H), Merton → Tascomi (browser-only), Stockport → Idox (Recipe C), Camden → cosmetically NEC (/NECSWS/ paths) but still PE underneath. Re-detect the vendor per council; don't assume NEC.

Recipe E — Ocella (/OcellaWeb/planningSearch)

Niche, plain server-rendered pages, and completely stateless — no session, CSRF, keyVal, or Referer required anywhere (keep a jar + browser UA + pacing as hygiene). $BASE = e.g. https://www1.arun.gov.uk/aplanning/OcellaWeb.

1. Shortcut — with an exact reference, skip search entirely: GET $BASE/planningDetails?reference=<REF>&from=planningSearch (raw ref with slashes works as-is). PlanIt's applics url field for Ocella councils is this deep link ready-made.
2. Otherwise POST $BASE/planningSearch with reference=<REF>&action=Search (other fields may be empty). Exact reference only — broad searches (e.g. location-only) silently re-render the empty form with no error. Extract detail links: grep -oE 'planningDetails\?reference=[^"]*'. Quirk: the postcode field is dotted (OcellaPlanningSearch.postcode); form dates are dd-mm-yy (8 chars).
3. Documents list: GET $BASE/showDocuments?reference=<REF>&module=pl (the detail page presents this as a POST form, but plain GET works; optional &filterBy=TYPE). Each row carries category, date (dd-mm-yy) and description — use them to label files.
4. Download: GET each viewDocument?file=…&module=pl href verbatim from showDocuments (don't construct: file is a URL-encoded server-side Windows path, dv_pl_files%5C<REF-with-underscores>%5C<filename>, spa

…(truncated)