# Test Processing Diff

> Test processing with diffing (local Docker)

- Skill: `fixmyberlin/test-processing-diff` (Agent Skill)
- Install (CLI): `npx skillmds@latest add fixmyberlin/test-processing-diff`
- Raw SKILL.md: https://api.skillmd.com/api/skills/fixmyberlin/test-processing-diff/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: DevOps & Infra
- Author: fixmyberlin (https://skillmd.com/u/fixmyberlin)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/fixmyberlin/test-processing-diff

---


# Test processing with diffing (local Docker)

Use after changing Lua/SQL under `processing/` (especially `processing/topics/`). With diffing enabled, the pipeline writes row-level diffs to `public."<table>_diff"`. Compare **reference** (baseline commit) vs **fixed** (your code) to see what your changes did.

## For agents: use the CLI (do not hand-craft `docker compose` env)

**Always** use `bun run processing` from **`app/`**. It builds **one line** that cds to the repo root in a subshell, sets env, and runs docker compose—your cwd is unchanged. **Interactive:** choose **Run command** to execute in this terminal, or **Show command** and paste the line. **Non-interactive / agents:** capture stdout and run that line in an environment with Docker.

**Defaults and flag checklist:** run `bun run processing -- --help` from `app/` for the full contract, injected skip defaults, and a copy-paste example.

**Default behavior:** with a TTY and **without** a complete non-interactive flag set, the script opens **interactive** prompts (Clack). `bun run processing` injects default skip flags; command-line skip/wait/download-url/osm2pgsql-log-level flags are applied without prompts. **Other** partial flags still require a full non-interactive set (you get a warning).

**Non-interactive (required for agents without a TTY, e.g. CI):** pass **every** required flag in one invocation. Required pieces:

- **Bbox:** `--preset <slug>` **or** both `--only-bbox` and `--diff-bbox` (optional: `--distinct-diff-bbox`, `--diff-bbox` with `--preset`).
- **`--diff-mode`** `off` | `previous` | `fixed` | `reference`.
- **Topics:** exactly one of `--all-daily-topics` (nightly only; usual for diff tests) **or** `--all-topics` (incl. weekly) **or** `--topics <csv>`.
- **Skips:** `--skip-download`, `--skip-unchanged`, `--skip-warm-cache` each with `0` or `1`. With `--skip-download 1`, also pass `--wait-fresh-data` `0` or `1`; with `--skip-download 0`, wait is forced off (flag optional).
- **Exactly one** of `--dry-run`, `--detach` (`-d`), or `--foreground` (controls the printed `docker compose` line; `--dry-run` and `--foreground` both use attached `up processing`).

**Learn the tool in this order:**

1. From **`app/`**: `bun run processing -- --help`.
2. **Agents / no TTY:** full flag set → capture stdout (the one-liner) → run that line in an environment with Docker (or instruct the user to paste it).
3. **Humans:** interactive generate → **Run command** or copy highlighted line → paste → Enter; reuse history and edit **`PROCESSING_DIFFING_MODE`** at the **end** of the env list for reference vs fixed.

**What you need to know (not how the script is implemented):**

- The printed command runs **docker compose** at the repository root (via subshell `cd`) so the **root `.env`** applies—the same file the app uses via `bun --env-file=../.env` from `app/`.
- Overrides are **per pasted command** — no need to `export` vars in the user’s shell.
- Ensure **`db` is healthy** before running the pasted line (`docker compose up -d db` from repo root if needed). The generate script does not start containers.
- On isolated worktree stacks, follow [tilda-geo-agent-workflow](../tilda-geo-agent-workflow/SKILL.md) for `.env.local` / `DEV_STACK_ID` rules; prefer the env from the printed `bun run processing` line over manual env.

Implementation: [`app/scripts/processing-generate-command/index.ts`](../../../app/scripts/processing-generate-command/index.ts). README: [`app/scripts/processing-generate-command/README.md`](../../../app/scripts/processing-generate-command/README.md). Script entry: [`app/package.json`](../../../app/package.json) (`processing`).

## Full batch examples (reference → fixed)

Use the **same** bbox, topics, and skip flags for reference and fixed; only change `--diff-mode` in the **generate** invocation (the printed line ends with `PROCESSING_DIFFING_MODE=…` last for easy edits).

**Generate reference line (inspect stdout):**

```bash
bun run processing -- \
  --preset xhain \
  --diff-mode reference \
  --all-daily-topics \
  --skip-download 1 \
  --skip-unchanged 0 \
  --skip-warm-cache 1 \
  --wait-fresh-data 0 \
  --foreground
```

**Generate fixed line:** same as above with `--diff-mode fixed`.

Paste each printed line to run the container. **Limited topics:** `--topics trafficSigns,parking`. **Include weekly (`landcover`):** `--all-topics` or list `landcover` in `--topics`.

**Common presets (`--preset <slug>`):** same bbox on **both** `PROCESS_ONLY_BBOX` and `PROCESSING_DIFFING_BBOX` unless you use `--distinct-diff-bbox` or separate `--only-bbox` / `--diff-bbox`. See `--help` for the full slug list.

| Slug                     | Coordinates                           | Notes                                   |
| ------------------------ | ------------------------------------- | --------------------------------------- |
| `xhain`                  | `13.380,52.488,13.418,52.503`         | Small                                   |
| `berlin` / `berlin-full` | `13.0883,52.3382,13.7611,52.6755`     | Large                                   |
| `bussonderstreifen`      | `13.38486,52.43778,13.38956,52.43959` | Interactive default for processing bbox |

`PROCESSING_DIFFING_BBOX` is **required** whenever diffing mode is not `off` (`processing/diffing/diffing.ts`). Effective diff area is the intersection of the two bboxes when both are set (`processing/diffing/diffing.ts`).

## Env: do not churn root `.env` for this loop

Keep Geofabrik OAuth, default extract URL (e.g. Berlin/Brandenburg), DB, and other secrets in **root** `.env`. For diff tests, **prefer generated env on the command line** instead of editing `.env`. To override the extract URL once, use **`--download-url`** on the CLI (not prompted interactively). See [`.env.example`](../../../.env.example).

## Git workflow (baseline → new code)

1. **Save work:** clean tree, temp commit, or stash.
2. **Baseline:** `git checkout <commit-before-changes>`.
3. **Reference run:** generate with `--diff-mode reference`, paste and run the printed line.
4. Fix failures if needed; re-run the same pasted command after fixes.
5. **Your branch:** `git checkout <branch-with-changes>`.
6. **Fixed run** — same flags except **`--diff-mode fixed`** (or edit only `PROCESSING_DIFFING_MODE` on a reused line).

Do **not** change other diff-related flags between reference and fixed unless you mean to invalidate the comparison.

**Mapillary pseudo-tags:** With `reference`, processing always re-downloads `mapillary_coverage.csv`. With `fixed`, it reuses that file (no re-download) so `mapillary_coverage` diffs reflect Lua changes only. Run reference before fixed on the same Docker volume.

Detached: use `--detach` in the generate invocation; the printed line uses `docker compose up -d processing`; then `docker logs -f processing`.

## Review `*_diff` tables

After the **fixed** run, inspect **`public`** only:

```sql
SELECT table_name
FROM information_schema.tables
WHERE table_schema = 'public'
  AND table_name LIKE '%\_diff' ESCAPE ''
ORDER BY table_name;
```

Map topics → table names via topic Lua/SQL or `processing/utils/TableNames.lua`.

**MCP / DB:** Use the user’s Postgres MCP or `psql` with `.env` (`DATABASE_HOST`, `DATABASE_NAME`, `DATABASE_USER`, `DATABASE_PASSWORD`).

**Interpretation:**

- Diffs compare **`tags` JSON** vs the reference snapshot (`processing/README.md`).
- Large diff ⇒ possible regression or broad tag changes; tiny/empty ⇒ often in-bounds or out of bbox/topic scope.
- **Noise:** rerun reference + fixed with `--skip-unchanged 0` and stable bbox/topics if unsure.

```sql
SELECT COUNT(*) FROM public.mytable_diff;
SELECT * FROM public.mytable_diff LIMIT 50;
```

## Quick sanity: Lua unit tests

Run from `processing/`:

`bun run test`

Notes:

- This wrapper runs Docker+busted for the whole suite.
- Lua modules are loaded via dotted requires (for example `topics.parking.roads.helper.result_tags`).

## Lua heap profiling (performance / memory investigations)

Use when tuning osm2pgsql Lua (CSV cache size, per-way growth) — not for routine diff tests.

1. **Helper:** [`processing/topics/helper/memory_reporter.lua`](../../../processing/topics/helper/memory_reporter.lua) — copy the usage example at the bottom into the topic under investigation (e.g. `prepare_pseudo_tags_roads_bikelanes.lua`, `load_merged_pseudo_tags.lua`). Remove the wiring after the run.
2. **Enable for one run:** add to repo root **`.env`** (not `docker-compose.yml`):
   - `PT_MEMORY_REPORT=1`
   - optional `PT_MEMORY_REPORT_EVERY=100000`
3. Run processing as usual (`bun run processing` → paste line). Logs show `MEMORY: …` on stdout / `docker logs processing`.
4. **Context:** national pseudo-tag performance notes live on branch `processing-performance-csvs-settlements` (`processing/docs/roads-bikelanes-nightly-performance-handoff.md`).

Do not leave `memory_reporter` calls in committed prod topic code.

## Related docs

- `processing/README.md` — diffing modes, `PROCESS_ONLY_*`, `SKIP_UNCHANGED`
- `processing/utils/parameters.ts` — env names the container reads
- `app/scripts/processing-generate-command/README.md` — copy-paste workflow, compose from repo root, single root `.env`