ArcGIS Hub → PortalJS
Overview
Migrate an entire ArcGIS Hub open-data site into a PortalJS Arc portal in one pass.
Every Hub site is machine-readable — a DCAT-US catalog at /data.json, with every dataset
backed by an ArcGIS REST FeatureService — so migration is a harvest → export → convert →
publish → verify pipeline that runs almost fully automated on the operator's machine, no
server-side compute. The tooling is the reusable arcgis-to-portaljs migrator: input is one
Hub URL, output is a ready-to-deploy PortalJS catalog plus a parity report.
The skill is an orchestrator: it reuses the DCAT-US harvest from portaljs-migrate, the
ogr2ogr/tippecanoe/duckdb dual-tier conversion from portaljs-add-geo, and the bulk
Git-LFS → R2 push from portaljs-migrate. Its novel parts are the FeatureService REST export
loop (paged features, not just a link) and the source-vs-derived parity report.
Prerequisites
- A scaffolded PortalJS portal whose template ships
components/MapPreview.tsx and
components/GeoQuery.tsx (PR #1647 or later). Run portaljs-new-portal first if none.
- Native CLIs: GDAL (
ogr2ogr, ogrinfo), tippecanoe, duckdb (with spatial),
and jq. macOS: brew install gdal tippecanoe duckdb jq; Debian/Ubuntu:
apt-get install gdal-bin duckdb jq plus tippecanoe (apt or build from source); Windows via
WSL. The skill hard-stops with the install hint if any is missing.
- Arc credentials for the Git-LFS → R2 push (the token
portaljs-deploy resolves), or an OSS
self-hosted Giftless.
Instructions
The canonical, full step-by-step workflow is
.claude/commands/arcgis-to-portaljs.md
— the single source of truth. Read and follow it when executing. Summary:
- Gather input — Hub URL, portal directory, project slug, optional flags (
--limit,
--only, --dry-run, --namespace-mode). Interview if missing; never dead-end.
- Check native tools (
ogr2ogr, tippecanoe, duckdb + spatial, jq). Any missing →
print the per-OS install and stop.
- Validate the portal directory and confirm the geo showcase components exist.
- Harvest the Hub
/data.json (reuse the portaljs-migrate DCAT-US map) and classify each
item: vector (FeatureService), table, or non-data (web map / 3D / imagery → skipped).
- Export each vector layer through the ArcGIS REST
query API with resultOffset paging
(f=geojson, outSR=4326); fall back to keyset paging on transfer limits; accept a
customer File Geodatabase dump for very large layers.
- Convert each layer to the dual tier via the
portaljs-add-geo recipe (PMTiles +
GeoParquet); tabular items to Parquet. Preserve the native-CRS original.
- Publish — bulk Git-LFS track + one push to R2 through Giftless, then append dual-tier
datasets.json entries (upsert on (namespace, slug)).
- Write
arcgis-parity-report.md — record count, extent, attribute schema, and geometry
validity, source vs derived, per dataset, plus the migrated/skipped/failed accounting.
- Report the inventory, migrated datasets, R2 push, and parity summary.
Output
- Created:
data/<namespace>/<slug>.pmtiles, .parquet, and the original per vector
dataset (all LFS-tracked → R2); Parquet + original per table; arcgis-parity-report.md.
- Modified:
datasets.json (one dual-tier entry per vector dataset, one resource entry
per table); .gitattributes (LFS tracking).
- Verified: the parity report compares each derived artifact to the live FeatureService.
- Result:
/@<namespace>/<slug> renders <MapPreview> + <GeoQuery> for each vector
dataset with no page edits; the catalog lists everything migrated.
Error Handling
| Symptom |
Cause |
Fix |
MISSING_INPUT |
No Hub URL provided |
Pass the site root (e.g. https://hub-lewisville.opendata.arcgis.com) and retry. |
MISSING_TOOLS |
ogr2ogr/tippecanoe/duckdb/jq (or duckdb spatial) absent |
Print the per-OS install line and stop; re-run after installing. |
NOT_A_PORTAL |
Target dir has no datasets.json / geo components |
Run portaljs-new-portal first, then re-run. |
HARVEST_FAILED |
/data.json unreachable or not DCAT-US |
Confirm the site is an ArcGIS Hub and the feed loads in a browser. |
EXPORT_FAILED |
One FeatureService layer errored or hit a hard transfer cap |
Logged and skipped; try keyset paging or a customer FGDB dump for that layer. |
LFS_PUSH_FAILED |
Missing/expired Arc token or unset lfs.url |
Re-mint the JWT (see portaljs-deploy); confirm git config lfs.url. |
Examples
Example 1 — Migrate a City Hub (Lewisville)
/arcgis-to-portaljs https://hub-lewisville.opendata.arcgis.com slug=lewisville
Example 2 — Dry-run inventory + plan only (no writes)
/arcgis-to-portaljs https://streamwaterdata.co.uk --dry-run
Example 3 — Migrate a subset, one namespace per publisher
/arcgis-to-portaljs https://streamwaterdata.co.uk --only sewer-catchments,water-boundaries --namespace-mode owner
Resources
1---2name: arcgis-to-portaljs3description: Migrate a whole ArcGIS Hub site into a PortalJS Arc portal end-to-end. Harvests the Hub /data.json (DCAT-US) inventory, exports every FeatureService layer through the ArcGIS REST query API with resultOffset paging, converts each to the serverless dual tier (PMTiles render + GeoParquet query) with tabular items to Parquet, pushes everything to Cloudflare R2 via Git LFS, appends dual-tier datasets.json entries, and writes a source-vs-derived parity report. Use to move a City or sector ArcGIS Hub open-data portal onto PortalJS with no server-side compute.4license: MIT5---67# ArcGIS Hub → PortalJS89## Overview1011Migrate an entire **ArcGIS Hub** open-data site into a **PortalJS Arc** portal in one pass.12Every Hub site is machine-readable — a DCAT-US catalog at `/data.json`, with every dataset13backed by an ArcGIS REST FeatureService — so migration is a **harvest → export → convert →14publish → verify** pipeline that runs almost fully automated on the operator's machine, no15server-side compute. The tooling is the reusable `arcgis-to-portaljs` migrator: input is one16Hub URL, output is a ready-to-deploy PortalJS catalog plus a parity report.1718The skill is an orchestrator: it reuses the DCAT-US harvest from `portaljs-migrate`, the19`ogr2ogr`/`tippecanoe`/`duckdb` dual-tier conversion from `portaljs-add-geo`, and the bulk20Git-LFS → R2 push from `portaljs-migrate`. Its novel parts are the FeatureService REST export21loop (paged features, not just a link) and the source-vs-derived parity report.2223## Prerequisites2425- A scaffolded PortalJS portal whose template ships `components/MapPreview.tsx` and26 `components/GeoQuery.tsx` (PR #1647 or later). Run `portaljs-new-portal` first if none.27- Native CLIs: **GDAL** (`ogr2ogr`, `ogrinfo`), **tippecanoe**, **duckdb** (with `spatial`),28 and **jq**. macOS: `brew install gdal tippecanoe duckdb jq`; Debian/Ubuntu:29 `apt-get install gdal-bin duckdb jq` plus tippecanoe (apt or build from source); Windows via30 WSL. The skill hard-stops with the install hint if any is missing.31- Arc credentials for the Git-LFS → R2 push (the token `portaljs-deploy` resolves), or an OSS32 self-hosted Giftless.3334## Instructions3536The canonical, full step-by-step workflow is37[`.claude/commands/arcgis-to-portaljs.md`](https://github.com/datopian/portaljs/blob/main/.claude/commands/arcgis-to-portaljs.md)38— the single source of truth. Read and follow it when executing. Summary:39401. Gather input — Hub URL, portal directory, project slug, optional flags (`--limit`,41 `--only`, `--dry-run`, `--namespace-mode`). Interview if missing; never dead-end.422. Check native tools (`ogr2ogr`, `tippecanoe`, `duckdb` + `spatial`, `jq`). Any missing →43 print the per-OS install and stop.443. Validate the portal directory and confirm the geo showcase components exist.454. Harvest the Hub `/data.json` (reuse the `portaljs-migrate` DCAT-US map) and classify each46 item: vector (FeatureService), table, or non-data (web map / 3D / imagery → skipped).475. Export each vector layer through the ArcGIS REST `query` API with `resultOffset` paging48 (`f=geojson`, `outSR=4326`); fall back to keyset paging on transfer limits; accept a49 customer File Geodatabase dump for very large layers.506. Convert each layer to the dual tier via the `portaljs-add-geo` recipe (PMTiles +51 GeoParquet); tabular items to Parquet. Preserve the native-CRS original.527. Publish — bulk Git-LFS track + one push to R2 through Giftless, then append dual-tier53 `datasets.json` entries (upsert on `(namespace, slug)`).548. Write `arcgis-parity-report.md` — record count, extent, attribute schema, and geometry55 validity, source vs derived, per dataset, plus the migrated/skipped/failed accounting.569. Report the inventory, migrated datasets, R2 push, and parity summary.5758## Output5960- **Created:** `data/<namespace>/<slug>.pmtiles`, `.parquet`, and the original per vector61 dataset (all LFS-tracked → R2); Parquet + original per table; `arcgis-parity-report.md`.62- **Modified:** `datasets.json` (one dual-tier entry per vector dataset, one resource entry63 per table); `.gitattributes` (LFS tracking).64- **Verified:** the parity report compares each derived artifact to the live FeatureService.65- **Result:** `/@<namespace>/<slug>` renders `<MapPreview>` + `<GeoQuery>` for each vector66 dataset with no page edits; the catalog lists everything migrated.6768## Error Handling6970| Symptom | Cause | Fix |71| --- | --- | --- |72| `MISSING_INPUT` | No Hub URL provided | Pass the site root (e.g. `https://hub-lewisville.opendata.arcgis.com`) and retry. |73| `MISSING_TOOLS` | `ogr2ogr`/`tippecanoe`/`duckdb`/`jq` (or duckdb `spatial`) absent | Print the per-OS install line and stop; re-run after installing. |74| `NOT_A_PORTAL` | Target dir has no `datasets.json` / geo components | Run `portaljs-new-portal` first, then re-run. |75| `HARVEST_FAILED` | `/data.json` unreachable or not DCAT-US | Confirm the site is an ArcGIS Hub and the feed loads in a browser. |76| `EXPORT_FAILED` | One FeatureService layer errored or hit a hard transfer cap | Logged and skipped; try keyset paging or a customer FGDB dump for that layer. |77| `LFS_PUSH_FAILED` | Missing/expired Arc token or unset `lfs.url` | Re-mint the JWT (see `portaljs-deploy`); confirm `git config lfs.url`. |7879## Examples8081### Example 1 — Migrate a City Hub (Lewisville)8283```84/arcgis-to-portaljs https://hub-lewisville.opendata.arcgis.com slug=lewisville85```8687### Example 2 — Dry-run inventory + plan only (no writes)8889```90/arcgis-to-portaljs https://streamwaterdata.co.uk --dry-run91```9293### Example 3 — Migrate a subset, one namespace per publisher9495```96/arcgis-to-portaljs https://streamwaterdata.co.uk --only sewer-catchments,water-boundaries --namespace-mode owner97```9899## Resources100101- Full workflow: [`.claude/commands/arcgis-to-portaljs.md`](https://github.com/datopian/portaljs/blob/main/.claude/commands/arcgis-to-portaljs.md)102- REST export, classification, parity, and phasing details: [`references/reference.md`](references/reference.md)103- Related skills: `portaljs-migrate`, `portaljs-add-geo`, `portaljs-add-dataset`, `portaljs-deploy`104- ArcGIS REST query API: <https://developers.arcgis.com/rest/services-reference/enterprise/query-feature-service-layer/> · tippecanoe: <https://github.com/felt/tippecanoe> · DuckDB spatial: <https://duckdb.org/docs/extensions/spatial>
gabrielmoreira