Add Static Dataset

Add Static Dataset

Adds a new static dataset folder to tilda-static-data and uses the tilda-geo upload pipeline to process it. In tilda-geo, app/scripts/StaticDatasets/geojson is a symlink and must point at the matching tilda-static-data worktree before editing data files.

When to Use

• User wants to add a new GeoJSON dataset
• User mentions adding static data, geojson files, "uploads", or datasets
• User provides a geojson file path and wants it integrated

Required Information

Extract from user prompt or ask if missing:

1. Group folder (e.g., region-berlin, region-bb) - parent folder name
2. Sub-folder name (e.g., berlin-bezirke) - dataset folder name (must be valid slug). Must include region/group prefix: for region-berlin use berlin-<name>, not just <name>
3. GeoJSON file path - absolute path to source file on disk
4. Regions - array of region slugs (e.g., ['infravelo'], ['berlin'])
5. Optional: transformation needs, style preferences, category, attribution, license

Process

0. Prepare tilda-static-data worktree and symlink

Static dataset files live in the sibling repo tilda-static-data, not in tilda-geo.

• If the task only reads existing static data, the regular symlink to ../../../../tilda-static-data/geojson is enough.
• If the task adds or edits dataset files, create a matching tilda-static-data worktree and relink geojson there.

For a branch named my-branch:

cd ../tilda-static-data
git worktree add ../tilda-static-data--my-branch my-branch

cd ../tilda-geo--my-branch/app
rm -f scripts/StaticDatasets/geojson
ln -s ../../../../tilda-static-data--my-branch/geojson scripts/StaticDatasets/geojson

Do not edit app/scripts/StaticDatasets/geojson when it points at the regular shared ../tilda-static-data checkout.

1. Create Folder Structure

app/scripts/StaticDatasets/geojson/<GROUP_FOLDER>/<SUB_FOLDER>/

Important: /geojson is a symlink. Before creating folders, ensure it points at the matching tilda-static-data--my-branch worktree.

Create directory if group folder doesn't exist. Ensure sub-folder name follows naming convention (see Required Information above).

2. Prepare GeoJSON file

Run these in order on committed data files. Never reproject, round, or pretty-print coordinates in transform.ts — use ogr2ogr (2.2–2.3) and oxfmt (2.4) on the files instead. Host GDAL 3.8+ (brew install gdal); not required for bun run dev.

2.1 Move

Move source file to:

app/scripts/StaticDatasets/geojson/<GROUP_FOLDER>/<SUB_FOLDER>/<FILENAME>.geojson

This path should resolve into ../tilda-static-data--my-branch/geojson/....

2.2 CRS → EPSG:4326

Check: Inspect the first coordinate pair after moving. For Germany, expect roughly lon 5–15, lat 47–55. Web Mercator (EPSG:3857) uses large meter values (e.g. ~1.5e6).

Fix (only if not WGS84): Reproject in place:

cd app/scripts/StaticDatasets/geojson/<GROUP_FOLDER>/<SUB_FOLDER>
ogr2ogr -f GeoJSON -t_srs EPSG:4326 -lco COORDINATE_PRECISION=8 \
  <FILENAME>.geojson.tmp <FILENAME>.geojson \
  && mv <FILENAME>.geojson.tmp <FILENAME>.geojson

2.3 Coordinate precision (max 8 decimals)

Check: Coordinates should have at most 8 decimal places (matches upload pipeline and API exports).

Fix (only if more than 8): Round in place (file must already be EPSG:4326):

cd app/scripts/StaticDatasets/geojson/<GROUP_FOLDER>/<SUB_FOLDER>
ogr2ogr -f GeoJSON -lco COORDINATE_PRECISION=8 \
  <FILENAME>.geojson.tmp <FILENAME>.geojson \
  && mv <FILENAME>.geojson.tmp <FILENAME>.geojson

Upload runs validateProjection again on WGS84 bounds.

2.4 Format (oxfmt)

From app/ (skill / agents only — devs use format-on-save in the editor):

bun run format-static-datasets-geojson -- \
  scripts/StaticDatasets/geojson/<GROUP_FOLDER>/<SUB_FOLDER>/*.{json,geojson}

2.5 Gzip (if large)

Check: Uncompressed .geojson size after formatting.

• > 6 MiB (6 × 1024² bytes): compress so the folder ships only the archive:

cd app/scripts/StaticDatasets/geojson/<GROUP_FOLDER>/<SUB_FOLDER>
gzip -9 -f <FILENAME>.geojson

gzip replaces the file with <FILENAME>.geojson.gz and removes the plain .geojson (-9 max compression, -f overwrite).

3. Create transform.ts (if needed)

Only if transformation required (not for CRS or coordinate precision — Step 2 only). Use helpers from app/scripts/StaticDatasets/geojson/_utils:

• transformUtils.ts - property transformations
• defaultLayerStyles.ts - default styling helpers
• translateUtils.ts - translation helpers

Example:

import { FeatureCollection } from 'geojson'

export const transform = (data: FeatureCollection) => {
  // Use helper functions from _utils when possible
  return data
}

4. Create meta.ts

Critical: Research similar datasets in same group folder first.

Before writing meta.ts:

1. List all datasets in <GROUP_FOLDER>/ directory
2. Read 2-3 similar meta.ts files (similar geometry type, similar purpose)
3. Infer patterns:
   • Category from similar datasets
   • Attribution patterns
   • License conventions
   • Style patterns
   • Inspector settings
4. If unclear, ask user to confirm or provide missing data

Required fields (from app/scripts/StaticDatasets/types.ts):

• regions: RegionSlug[] (required)
• public: boolean (required)
• dataSourceType: 'local' (required)
• attributionHtml: string (required)
• configs: Array with at least one config (required)

Optional on export const data:

• dataUpdatedNote: string
• dataSourceMarkdown: string
• licence: License type (see types.ts)
• licenceOsmCompatible: 'licence' | 'waiver' | 'no'

Config required fields:

• name: string
• inspector: { enabled: boolean, ... } or { enabled: false }
• layers: Layer[] (required)

Config optional fields:

• category: string | null — grouping key; titles/order are managed in Admin → Statische Datensatz-Kategorien (DB).
• description: string
• legends: Legend[]

Style/Legend: If not specified, make best guess:

• Use defaultLayerStyles() from _utils/defaultLayerStyles.ts for simple cases
• For polygons: fill + outline
• For lines: colored line with width
• For points: circle markers
• Create appropriate legend entries

Stripe / Schraffur (fill-pattern): For diagonal stripes over polygons, reuse the existing map sprite image id stripe_texture (same hatch as private Parkflächen in generated parking_areas; listed in app/public/map-style/sprite.json). Do not add a new sprite. Stack a second fill layer above the solid fill: fill-color 'rgba(0, 0, 0, 0)', fill-opacity around 0.5 (tune as needed), fill-pattern 'stripe_texture'. Skip fill-outline-color on the stripe layer if the base fill already defines an outline. Use a filter on the stripe layer when only some features should be hatched; if exports mix boolean true and string 'true', either use an any / match filter in meta.ts or normalize values in transform.ts (step 3) so the filter stays simple.

Example structure:

import { MetaData } from '../../../types'
import { defaultLayerStyles } from '../../_utils/defaultLayerStyles'

export const data: MetaData = {
  regions: ['infravelo'], // From user or infer from group folder
  public: true,
  dataSourceType: 'local',
  attributionHtml: 'Source Name', // Ask if unclear
  licence: 'DL-DE/ZERO-2.0', // Infer from similar datasets
  configs: [
    {
      name: 'Dataset Name',
      categoryKey: 'berlin/misc', // Infer from similar datasets
      inspector: { enabled: false }, // Default unless specified
      layers: defaultLayerStyles(), // Or custom layers
    },
  ],
}

5. Verify Command

Check app/scripts/StaticDatasets/updateStaticDatasets.ts for correct params:

• --folder-filter=<SUB_FOLDER_NAME> (matches full path, so sub-folder name works)
• --env=dev (or staging/production; omit to be prompted)

One-click command from app/ (replace <SUB_FOLDER_NAME> with actual folder name; static-datasets-update uses the repository root .env via bun --env-file=../.env, same as bun run dev):

bun run static-datasets-update -- --folder-filter=<SUB_FOLDER_NAME> --env=dev

Note: updateDownloadSources.ts is for WFS downloads (requires downloadConfig.ts). Use updateStaticDatasets.ts for local GeoJSON files.

6. Verify TypeScript Compilation

Always run this after creating or modifying TypeScript files (meta.ts, transform.ts, or any imports). From app/ (same as Step 5):

bun run type-check-deploy

This temporarily removes the geojson symlink, runs TypeScript type-checking, and restores the symlink. It simulates the Docker build environment where symlinks aren't available, ensuring your code will compile correctly during builds.

Validation

Before completing:

1. ✅ Folder structure created
2. ✅ GeoJSON prepared (Step 2.1–2.5: moved, EPSG:4326, ≤8 decimals, oxfmt, gzip if >6 MiB)
3. ✅ transform.ts only if needed — never for CRS (Step 2); meta.ts / transform.ts via format-on-save or format-static-datasets-geojson
4. ✅ meta.ts follows type structure (check with TypeScript)
5. ✅ Similar datasets in group folder reviewed for patterns
6. ✅ Command verified and provided as one-click action
7. ✅ bun run type-check-deploy run successfully (see Step 6)

References

• User-facing request template (DE, GitHub issue style): ANFRAGE-STATISCHER-DATENSATZ.md
• Types: app/scripts/StaticDatasets/types.ts
• Examples: app/scripts/StaticDatasets/geojson/region-berlin/*/meta.ts
• Utils: app/scripts/StaticDatasets/geojson/_utils/
• Docs: docs/Features-Parameter-Deeplinks.md
• Update script: app/scripts/StaticDatasets/updateStaticDatasets.ts