Test Processing Diff

Test processing with diffing (local Docker)

Use after changing Lua/SQL under processing/ (especially processing/topics/). With diffing enabled, the pipeline writes row-level diffs to public."<table>_diff". Compare reference (baseline commit) vs fixed (your code) to see what your changes did.

For agents: use the CLI (do not hand-craft docker compose env)

Always use bun run processing from app/. It builds one line that cds to the repo root in a subshell, sets env, and runs docker compose—your cwd is unchanged. Interactive: choose Run command to execute in this terminal, or Show command and paste the line. Non-interactive / agents: capture stdout and run that line in an environment with Docker.

Defaults and flag checklist: run bun run processing -- --help from app/ for the full contract, injected skip defaults, and a copy-paste example.

Default behavior: with a TTY and without a complete non-interactive flag set, the script opens interactive prompts (Clack). bun run processing injects default skip flags; command-line skip/wait/download-url/osm2pgsql-log-level flags are applied without prompts. Other partial flags still require a full non-interactive set (you get a warning).

Non-interactive (required for agents without a TTY, e.g. CI): pass every required flag in one invocation. Required pieces:

• Bbox: --preset <slug> or both --only-bbox and --diff-bbox (optional: --distinct-diff-bbox, --diff-bbox with --preset).
• --diff-mode off | previous | fixed | reference.
• Topics: exactly one of --all-daily-topics (nightly only; usual for diff tests) or --all-topics (incl. weekly) or --topics <csv>.
• Skips: --skip-download, --skip-unchanged, --skip-warm-cache each with 0 or 1. With --skip-download 1, also pass --wait-fresh-data 0 or 1; with --skip-download 0, wait is forced off (flag optional).
• Exactly one of --dry-run, --detach (-d), or --foreground (controls the printed docker compose line; --dry-run and --foreground both use attached up processing).

Learn the tool in this order:

1. From app/: bun run processing -- --help.
2. Agents / no TTY: full flag set → capture stdout (the one-liner) → run that line in an environment with Docker (or instruct the user to paste it).
3. Humans: interactive generate → Run command or copy highlighted line → paste → Enter; reuse history and edit PROCESSING_DIFFING_MODE at the end of the env list for reference vs fixed.

What you need to know (not how the script is implemented):

• The printed command runs docker compose at the repository root (via subshell cd) so the root .env applies—the same file the app uses via bun --env-file=../.env from app/.
• Overrides are per pasted command — no need to export vars in the user’s shell.
• Ensure db is healthy before running the pasted line (docker compose up -d db from repo root if needed). The generate script does not start containers.
• On isolated worktree stacks, follow tilda-geo-agent-workflow for .env.local / DEV_STACK_ID rules; prefer the env from the printed bun run processing line over manual env.

Implementation: app/scripts/processing-generate-command/index.ts. README: app/scripts/processing-generate-command/README.md. Script entry: app/package.json (processing).

Full batch examples (reference → fixed)

Use the same bbox, topics, and skip flags for reference and fixed; only change --diff-mode in the generate invocation (the printed line ends with PROCESSING_DIFFING_MODE=… last for easy edits).

Generate reference line (inspect stdout):

bun run processing -- \
  --preset xhain \
  --diff-mode reference \
  --all-daily-topics \
  --skip-download 1 \
  --skip-unchanged 0 \
  --skip-warm-cache 1 \
  --wait-fresh-data 0 \
  --foreground

Generate fixed line: same as above with --diff-mode fixed.

Paste each printed line to run the container. Limited topics: --topics trafficSigns,parking. Include weekly (landcover): --all-topics or list landcover in --topics.

Common presets (--preset <slug>): same bbox on both PROCESS_ONLY_BBOX and PROCESSING_DIFFING_BBOX unless you use --distinct-diff-bbox or separate --only-bbox / --diff-bbox. See --help for the full slug list.

PROCESSING_DIFFING_BBOX is required whenever diffing mode is not off (processing/diffing/diffing.ts). Effective diff area is the intersection of the two bboxes when both are set (processing/diffing/diffing.ts).

Env: do not churn root .env for this loop

Keep Geofabrik OAuth, default extract URL (e.g. Berlin/Brandenburg), DB, and other secrets in root .env. For diff tests, prefer generated env on the command line instead of editing .env. To override the extract URL once, use --download-url on the CLI (not prompted interactively). See .env.example.

Git workflow (baseline → new code)

1. Save work: clean tree, temp commit, or stash.
2. Baseline: git checkout <commit-before-changes>.
3. Reference run: generate with --diff-mode reference, paste and run the printed line.
4. Fix failures if needed; re-run the same pasted command after fixes.
5. Your branch: git checkout <branch-with-changes>.
6. Fixed run — same flags except --diff-mode fixed (or edit only PROCESSING_DIFFING_MODE on a reused line).

Do not change other diff-related flags between reference and fixed unless you mean to invalidate the comparison.

Mapillary pseudo-tags: With reference, processing always re-downloads mapillary_coverage.csv. With fixed, it reuses that file (no re-download) so mapillary_coverage diffs reflect Lua changes only. Run reference before fixed on the same Docker volume.

Detached: use --detach in the generate invocation; the printed line uses docker compose up -d processing; then docker logs -f processing.

Review *_diff tables

After the fixed run, inspect public only:

SELECT table_name
FROM information_schema.tables
WHERE table_schema = 'public'
  AND table_name LIKE '%\_diff' ESCAPE ''
ORDER BY table_name;

Map topics → table names via topic Lua/SQL or processing/utils/TableNames.lua.

MCP / DB: Use the user’s Postgres MCP or psql with .env (DATABASE_HOST, DATABASE_NAME, DATABASE_USER, DATABASE_PASSWORD).

Interpretation:

• Diffs compare tags JSON vs the reference snapshot (processing/README.md).
• Large diff ⇒ possible regression or broad tag changes; tiny/empty ⇒ often in-bounds or out of bbox/topic scope.
• Noise: rerun reference + fixed with --skip-unchanged 0 and stable bbox/topics if unsure.

SELECT COUNT(*) FROM public.mytable_diff;
SELECT * FROM public.mytable_diff LIMIT 50;

Quick sanity: Lua unit tests

Run from processing/:

bun run test

Notes:

• This wrapper runs Docker+busted for the whole suite.
• Lua modules are loaded via dotted requires (for example topics.parking.roads.helper.result_tags).

Lua heap profiling (performance / memory investigations)

Use when tuning osm2pgsql Lua (CSV cache size, per-way growth) — not for routine diff tests.

1. Helper: processing/topics/helper/memory_reporter.lua — copy the usage example at the bottom into the topic under investigation (e.g. prepare_pseudo_tags_roads_bikelanes.lua, load_merged_pseudo_tags.lua). Remove the wiring after the run.
2. Enable for one run: add to repo root .env (not docker-compose.yml):
   • PT_MEMORY_REPORT=1
   • optional PT_MEMORY_REPORT_EVERY=100000
3. Run processing as usual (bun run processing → paste line). Logs show MEMORY: … on stdout / docker logs processing.
4. Context: national pseudo-tag performance notes live on branch processing-performance-csvs-settlements (processing/docs/roads-bikelanes-nightly-performance-handoff.md).

Do not leave memory_reporter calls in committed prod topic code.

Related docs

• processing/README.md — diffing modes, PROCESS_ONLY_*, SKIP_UNCHANGED
• processing/utils/parameters.ts — env names the container reads
• app/scripts/processing-generate-command/README.md — copy-paste workflow, compose from repo root, single root .env