Merge Divergent Tox

Merging Divergent .tox / .toe Binaries

Git cannot three-way-merge a .tox. It reports a conflict and offers you "ours or theirs" -- a choice that silently discards one side's work.

This turns the binary into a reviewable text diff. Phases 0-2 need no TouchDesigner at all and carry no risk. In practice they often answer the whole question, so do not skip them to get to the interesting part.

The rule that governs everything here

This is a THREE-way merge, not a two-way one. Both sides changed the file from a common ancestor. If you compare only ours against theirs, a difference tells you nothing about who moved: an operator present in theirs and absent in ours may be their addition or our deletion, and those demand opposite actions. Always diff each side against the BASE.

Phase 0 -- Triage (no tools)

1. Enumerate every conflicting binary, not just the ones you were asked about.

   git diff --name-only --diff-filter=U                      # active conflicts
   git diff --stat <base>..<theirs> -- '*.tox' '*.toe'       # everything that diverged
   

   A file nobody mentioned is exactly the one that breaks later.

2. Find the true per-file content base. git merge-base is often wrong -- history rewrites and re-committed blobs move it. Verify per file; if the repo's history was rewritten, compare commit SUBJECTS rather than SHAs to find which commits are genuinely absent.

3. Answer the product questions BEFORE opening TouchDesigner. If one side deleted a subsystem the other side extended, that is a product decision, not a merge decision. Resolving it can collapse the hardest files to "take ours" and end the job. Ask the human. This is the highest-leverage step in the skill and it costs nothing.

Phase 1 -- Extract all three sides (git, binary-safe)

Use Bash/Git Bash, not PowerShell -- PowerShell redirection corrupts binary bytes.

W=/c/tmp/tox-merge; mkdir -p $W/{ours,theirs,base,work}; cd $W
git -C <REPO> cat-file blob HEAD:<path>       > ours/<name>.tox
git -C <REPO> cat-file blob origin/dev:<path> > theirs/<name>.tox
git -C <REPO> cat-file blob <baserev>:<path>  > base/<name>.tox

Integrity gate -- ABORT if either fails:

git -C <REPO> cat-file -s HEAD:<path>   # must equal the on-disk size
xxd -l 6 ours/<name>.tox                # must be: 0000 0001 3130

A mismatch means a smudge/eol filter mangled the extraction and every conclusion downstream is garbage.

Phase 2 -- Offline operator inventory with toeexpand

TouchDesigner ships toeexpand.exe in its bin/ folder. It expands a binary into an ASCII tree: one .n file per operator (older files use .init), a subdirectory per COMP. It works on .tox, not only .toe -- the docs mention only .toe, but Embody's shipping code runs it on .tox.

TDBIN="/c/Program Files/Derivative/TouchDesigner.<VER>/bin"
for side in base ours theirs; do
  ( cd $side && "$TDBIN/toeexpand.exe" <name>.tox )
  ( cd $side/<name>.tox.dir && find . \( -name '*.n' -o -name '*.init' \) \
      | sed 's/\.n$//;s/\.init$//' | sort ) > ../work/${side}_<name>.inv
done

comm -13 work/base_<name>.inv work/theirs_<name>.inv   # THEY added
comm -23 work/base_<name>.inv work/theirs_<name>.inv   # THEY removed
comm -13 work/base_<name>.inv work/ours_<name>.inv     # WE added
comm -23 work/base_<name>.inv work/ours_<name>.inv     # WE removed

toeexpand's exit code is unreliable -- it returns 1 on success. Judge by whether the .dir appeared, never by $?.

Read the inventory before anything else. It answers, for free:

• Is their delta all one subsystem? If so and that subsystem is being retired, the file is "take ours" and you are done.
• Are the added operators real work, or baked runtime clones? Extensions that generate per-machine children (ndiin_*, per-node copies) bake those children into whatever .tox was saved on that machine. They look like a large, impressive delta and are pure garbage from someone else's cluster. A run of numbered siblings is the tell.
• Did both sides delete the same thing? Convergent deletion is not a conflict.

.n files also carry parameter values and expressions -- grep them to decide much of the merge offline:

grep -rl "<term>" theirs/<name>.tox.dir          # which ops mention it
cat theirs/<name>.tox.dir/<path>/<op>.parm       # that op's parameters
cat ours/<name>.tox.dir/<comp>.cparm             # custom parameter table

Checking the .cparm against grep -oE "par\.[A-Z][A-Za-z0-9]*" on the extension source is a cheap way to catch code referencing a parameter that does not exist.

Phase 3 -- Sandbox (only if Phase 2 left real work)

Needed only when you must inspect wiring/panel state the inventory cannot settle, or when you must APPLY a change.

Never host the sandbox in a production project

TD extensions run at instantiation -- before onStart, before any init hook, gated behind nothing. A project whose extensions open a database, rewrite a config pointer, or notify peers has already caused damage by the time the network is visible. Relative paths in that code resolve against the process CWD, so copying the .toe elsewhere does not protect you, and if those writes are gitignored there is no recovery path.

Safe Mode is not a workaround. It guarantees nodes do not cook; it says nothing about extensions being instantiated.

Host instead in a purpose-made empty .toe that:

• lives outside every live project's git root (a TD instance rooted at a live repo can make tooling deploy config into it and clobber .mcp.json);
• is itself a git root (git init) so per-root tooling config lands there;
• uses an Envoy/MCP port clear of every live instance.

Drive it from a second agent session rooted at the sandbox -- per-root instance registries mean a session rooted elsewhere cannot see it.

Load contained

Create the host COMP, disable cooking, then load -- in that order, in one call, so nothing cooks in between:

h = op('/').create(baseCOMP, 'toxmerge_theirs')
h.allowCooking = False
child = h.loadTox(r'C:\tmp\tox-merge\theirs\name.tox')

Load both sides simultaneously (_ours and _theirs, offset on the grid). That is what makes applying possible: you copy operators between them.

Accepted tradeoff: with cooking off, "the outputs of the nodes inside are empty and undefined". Authored state exports fine; a derived DAT's content may export empty. Never read that as a deletion.

Phase 4 -- Export and diff (only when Phase 3 was necessary)

export_network(root_path=..., embed_all=True, include_dat_content=True)

• embed_all=True is mandatory. Without it, nested .tox children export as empty shells -- and a component with nested children is exactly the kind you are merging.
• Do not use a live-vs-disk diff tool for this. Those compare in-memory against an on-disk export and typically omit embed_all.
• Export one side twice first to establish the format-noise floor. Ignore volatile header keys (build, generator, timestamps, source path).

Phase 5 -- Apply, converge, verify

Classify every difference against the BASE into: theirs-added (port it), ours-added (keep), both-changed (decide, and say why), convergent (no action). Apply into the ours side, re-export, re-diff, and repeat until only intended differences remain. Save with COMP.save(), then reload the saved file and diff once more -- a round-trip gate.

What this technique CANNOT see

A clean diff is not proof of equivalence. Invisible to it:

Budget an eyeball pass in the TD UI over both loaded COMPs before accepting any merge of a UI or networking component.

What Embody already does for you

Phases 1, 2 and 4 are mechanical, and two of them have shipping tools:

• diff_tdn compares TDXN networks and is the right instrument once both sides are loaded or exported. Pair it with export_network(embed_all=True) per Phase 4.
• Externalizing to TDXN in the first place is the real fix. A component tracked as .tdxn is YAML on disk: git three-way-merges it like any other text file and this entire skill becomes unnecessary for it. If you are merging the same .tox by hand twice, externalize it as TDXN instead.

The judgement -- which side wins, and whether a subsystem is being retired -- stays human.