PortalJS — Migrate
Overview
Harvest datasets from an external open-data platform into a PortalJS portal, or
push them into a CKAN instance. Every source reads into one canonical shape —
{ slug, namespace, name, description, resources[] } — then writes to the target
from that shape, so adding a source covers every target automatically.
Supported sources: CKAN, a DCAT-US /data.json catalog (DKAN, ArcGIS Hub,
data.gov), a DCAT / DCAT-AP RDF feed, Socrata, OpenDataSoft, or an ArcGIS
FeatureServer/MapServer. Supported targets: a static PortalJS catalog
(datasets.json, resources linked by URL or downloaded into Cloudflare R2) or a
CKAN instance over its write API.
This is the copy-into-the-portal path, the inverse of /portaljs-connect-ckan
(which reads the source live at build time): a one-time, re-runnable snapshot so
the portal stands alone and needs no backend.
Prerequisites
- An existing PortalJS portal for the static target:
datasets.json,
package.json, and pages/[owner]/[slug].tsx must exist (the portaljs-catalog
template).
- Node 18+ and npm, with portal dependencies installed (
npm install).
- Network access to the source platform's API or feed URL.
- For a CKAN target: a write API key in
CKAN_API_KEY — never hardcoded.
- For
download copy mode: git-lfs and a PortalJS Arc account (or a
self-hosted Giftless endpoint) to push resource files to R2.
- For
dcat-rdf sources: lib/metadata/dcat-harvest.ts and dcat-profiles.ts
(copy from examples/portaljs-catalog/lib/metadata/ if missing), run via npx tsx.
Instructions
Full step-by-step workflow:
.claude/commands/portaljs-migrate.md.
- Gather input — source type, source URL, target (
static/ckan), portal
directory, copy mode (link/download), filters. Interview instead of erroring.
- Validate the target — confirm the static catalog files exist, or that the CKAN
URL and API key authenticate.
- Detect the source type from the URL (or
--source) and verify reachability.
- Read the source into canonical dataset entries, per the source's field table in
the reference doc, applying the metadata hygiene rules (license sentinel — never
fabricate CC-BY; description sanitizer + fallback chain; source dates preserved
with
migratedAt separate; cleanTitle display-name cleanup with slugs from the
raw title; category from a meaningful theme).
- For a static target, resolve resource paths by copy mode:
link keeps source
URLs; download copies files into the repo under Git LFS, pushed to R2.
- Print a dry-run preview; stop if
--dry-run was passed.
- Write
datasets.json (upsert or --replace), or push packages/resources into
CKAN over its action API.
- Verify —
npm run build (static) or re-query package_search (CKAN).
- Report datasets migrated, namespaces touched, and next steps.
Output
Static target: an updated datasets.json (upserted on (namespace, slug)),
optionally with resource files in Git LFS served from Cloudflare R2, plus a
passing npm run build. CKAN target: created/updated packages and resources;
per-dataset failures are logged and skipped, not fatal.
Error Handling
| Symptom |
Cause |
Fix |
"not a portaljs-catalog template" |
datasets.json missing |
Confirm the portal directory, or run /portaljs-new-portal |
| CKAN target rejects every write (403) |
Key missing, expired, or lacks org permission |
Set a valid CKAN_API_KEY; stop rather than partially migrating |
| Source URL matches no known format |
Auto-detection failed |
Pass --source explicitly (ckan, dcat, dcat-rdf, socrata, ods, arcgis) |
| DCAT-RDF fetch returns HTML, not a feed |
URL is a portal page, not the feed |
Follow its <link rel="alternate"> href, or use the direct feed URL |
| Build fails after write |
Malformed entry (missing slug/namespace/name) |
Inspect /tmp/portaljs-migrate-build.log, fix, rebuild |
download mode push fails |
git-lfs missing, or no Arc/Giftless token |
Install git-lfs; mint a token via the Arc API or self-hosted Giftless |
| Duplicate datasets after re-running |
--replace used unintentionally, or slug collisions |
Omit --replace to upsert; keep slugs unique per namespace |
Examples
Example 1 — CKAN to static catalog, link mode
/portaljs-migrate --source ckan --source-url https://demo.dev.datopian.com \
--target static --portal-dir . --copy-mode link
Upserts all packages from the CKAN demo instance into datasets.json, referencing
source resource URLs directly.
Example 2 — data.gov DCAT-US catalog, downloaded into R2
/portaljs-migrate --source dcat --source-url https://catalog.data.gov/data.json \
--target static --copy-mode download --org-filter epa-gov
Harvests one publisher and copies every resource file into the repo under Git
LFS, served from Cloudflare R2.
Example 3 — DCAT-AP RDF feed from a national portal
/portaljs-migrate --source dcat-rdf \
--source-url https://data.europa.eu/api/hub/search/catalog.jsonld \
--target static --dry-run
Fetches the JSON-LD feed through the RDF harvester and previews the canonical
datasets without writing anything.
Example 4 — CKAN to CKAN (platform-to-platform)
/portaljs-migrate --source ckan --source-url https://old-portal.example.org \
--target ckan --target-url https://new-portal.example.org --owner-org research
Pushes every package from the old CKAN instance into the new one, filed under
research.
Resources
1---2name: portaljs-migrate3description: Migrate (harvest) datasets between open-data platforms. Reads CKAN, a DCAT-US /data.json catalog (DKAN, ArcGIS Hub, data.gov), a DCAT / DCAT-AP RDF feed (JSON-LD, Turtle, or RDF/XML), Socrata, OpenDataSoft, or an ArcGIS FeatureServer, and writes them to a static PortalJS catalog or pushes them into a CKAN instance over its API. Use when moving datasets from an external open-data platform into a PortalJS portal, or bridging one CKAN instance to another.4license: MIT5---6
7# PortalJS — Migrate
8
9## Overview
10
11Harvest datasets from an external open-data platform into a PortalJS portal, or
12push them into a CKAN instance. Every source reads into one canonical shape —
13`{ slug, namespace, name, description, resources[] }` — then writes to the target
14from that shape, so adding a source covers every target automatically.
15
16Supported sources: CKAN, a DCAT-US `/data.json` catalog (DKAN, ArcGIS Hub,
17data.gov), a DCAT / DCAT-AP RDF feed, Socrata, OpenDataSoft, or an ArcGIS
18FeatureServer/MapServer. Supported targets: a static PortalJS catalog
19(`datasets.json`, resources linked by URL or downloaded into Cloudflare R2) or a
20CKAN instance over its write API.
21
22This is the copy-into-the-portal path, the inverse of `/portaljs-connect-ckan`
23(which reads the source live at build time): a one-time, re-runnable snapshot so
24the portal stands alone and needs no backend.
25
26## Prerequisites
27
28- An existing PortalJS portal for the static target: `datasets.json`,
29 `package.json`, and `pages/[owner]/[slug].tsx` must exist (the `portaljs-catalog`
30 template).
31- Node 18+ and npm, with portal dependencies installed (`npm install`).
32- Network access to the source platform's API or feed URL.
33- For a CKAN target: a write API key in `CKAN_API_KEY` — never hardcoded.
34- For `download` copy mode: `git-lfs` and a PortalJS Arc account (or a
35 self-hosted Giftless endpoint) to push resource files to R2.
36- For `dcat-rdf` sources: `lib/metadata/dcat-harvest.ts` and `dcat-profiles.ts`
37 (copy from `examples/portaljs-catalog/lib/metadata/` if missing), run via `npx tsx`.
38
39## Instructions
40
41Full step-by-step workflow:
42[`.claude/commands/portaljs-migrate.md`](https://github.com/datopian/portaljs/blob/main/.claude/commands/portaljs-migrate.md).
43
441. Gather input — source type, source URL, target (`static`/`ckan`), portal
45 directory, copy mode (`link`/`download`), filters. Interview instead of erroring.
462. Validate the target — confirm the static catalog files exist, or that the CKAN
47 URL and API key authenticate.
483. Detect the source type from the URL (or `--source`) and verify reachability.
494. Read the source into canonical dataset entries, per the source's field table in
50 the reference doc, applying the metadata hygiene rules (license sentinel — never
51 fabricate CC-BY; description sanitizer + fallback chain; source dates preserved
52 with `migratedAt` separate; `cleanTitle` display-name cleanup with slugs from the
53 raw title; `category` from a meaningful theme).
545. For a static target, resolve resource paths by copy mode: `link` keeps source
55 URLs; `download` copies files into the repo under Git LFS, pushed to R2.
566. Print a dry-run preview; stop if `--dry-run` was passed.
577. Write `datasets.json` (upsert or `--replace`), or push packages/resources into
58 CKAN over its action API.
598. Verify — `npm run build` (static) or re-query `package_search` (CKAN).
609. Report datasets migrated, namespaces touched, and next steps.
61
62## Output
63
64Static target: an updated `datasets.json` (upserted on `(namespace, slug)`),
65optionally with resource files in Git LFS served from Cloudflare R2, plus a
66passing `npm run build`. CKAN target: created/updated packages and resources;
67per-dataset failures are logged and skipped, not fatal.
68
69## Error Handling
70
71| Symptom | Cause | Fix |
72| --- | --- | --- |
73| "not a `portaljs-catalog` template" | `datasets.json` missing | Confirm the portal directory, or run `/portaljs-new-portal` |
74| CKAN target rejects every write (403) | Key missing, expired, or lacks org permission | Set a valid `CKAN_API_KEY`; stop rather than partially migrating |
75| Source URL matches no known format | Auto-detection failed | Pass `--source` explicitly (`ckan`, `dcat`, `dcat-rdf`, `socrata`, `ods`, `arcgis`) |
76| DCAT-RDF fetch returns HTML, not a feed | URL is a portal page, not the feed | Follow its `<link rel="alternate">` href, or use the direct feed URL |
77| Build fails after write | Malformed entry (missing `slug`/`namespace`/`name`) | Inspect `/tmp/portaljs-migrate-build.log`, fix, rebuild |
78| `download` mode push fails | `git-lfs` missing, or no Arc/Giftless token | Install `git-lfs`; mint a token via the Arc API or self-hosted Giftless |
79| Duplicate datasets after re-running | `--replace` used unintentionally, or slug collisions | Omit `--replace` to upsert; keep slugs unique per namespace |
80
81## Examples
82
83### Example 1 — CKAN to static catalog, link mode
84
85```bash
86/portaljs-migrate --source ckan --source-url https://demo.dev.datopian.com \
87 --target static --portal-dir . --copy-mode link
88```
89Upserts all packages from the CKAN demo instance into `datasets.json`, referencing
90source resource URLs directly.
91
92### Example 2 — data.gov DCAT-US catalog, downloaded into R2
93
94```bash
95/portaljs-migrate --source dcat --source-url https://catalog.data.gov/data.json \
96 --target static --copy-mode download --org-filter epa-gov
97```
98Harvests one publisher and copies every resource file into the repo under Git
99LFS, served from Cloudflare R2.
100
101### Example 3 — DCAT-AP RDF feed from a national portal
102
103```bash
104/portaljs-migrate --source dcat-rdf \
105 --source-url https://data.europa.eu/api/hub/search/catalog.jsonld \
106 --target static --dry-run
107```
108Fetches the JSON-LD feed through the RDF harvester and previews the canonical
109datasets without writing anything.
110
111### Example 4 — CKAN to CKAN (platform-to-platform)
112
113```bash
114/portaljs-migrate --source ckan --source-url https://old-portal.example.org \
115 --target ckan --target-url https://new-portal.example.org --owner-org research
116```
117Pushes every package from the old CKAN instance into the new one, filed under
118`research`.
119
120## Resources
121
122- Command source: [`.claude/commands/portaljs-migrate.md`](https://github.com/datopian/portaljs/blob/main/.claude/commands/portaljs-migrate.md)
123- Field mappings and troubleshooting: [`references/reference.md`](references/reference.md)
124- Related skills: `/portaljs-connect-ckan`, `/portaljs-add-dcat`,
125 `/portaljs-check-data-quality`, `/portaljs-define-schema`
126- CKAN Action API reference: https://docs.ckan.org/en/latest/api/
datopian