Gum Codegen Test Rigor

Gum Codegen: What Actually Proves a Fix Works

Glue's Gum codegen (gum-codegen skill) emits C# source text with no compile-time link to the real runtime types it targets. That's why weak verification repeatedly looks green while real bugs ship — this skill is the record of that pattern recurring, so the next fix doesn't restart at the bottom of this list.

Verification strength ladder — each level misses a different bug class

From weakest to strongest, all four were tried in sequence on the same fix before it actually worked:

1. String-matching generated source (generatedSource.ShouldContain/ShouldNotContain(...)). Proves text presence only. Misses everything about correctness.
2. Reflection-based member-contract checks (GumRuntimeMemberContractTests). Proves a named member exists on the contained type, but only for patterns the check explicitly matches (e.g. a ContainedXxx.Member regex). Missed a base-class (GraphicalUiElement.RenderableComponent) assignment entirely — wrong access shape, so the pattern never matched it, so it never looked.
3. Real compile against real assemblies (BuildSmoke *CreationSmokeTests / GumGeneratedCodeCompilesTests). Proves the generated code compiles. Says nothing about runtime behavior — code that compiles clean can still NRE the instant its constructor runs.
4. Real instantiate-and-execute. Actually construct the generated runtime the way production code does, and run it. This is the only level that catches a construction-time NullReferenceException. Nothing weaker than this can.

If a fix "passed" at one level and still broke in the field, the next debugging step is always "move up this ladder," not "add another test at the same level."

The sharper landmine: matching the real call path, not just reaching a level

Being at the top of the ladder isn't sufficient either — a test can compile-and-run and still miss the bug if it doesn't reproduce the exact parameters/state production code actually uses:

• A BuildSmoke sweep silently defaulted GumProjectSave.Version, so it always exercised the v2 branch and never touched the v3 code path it was supposedly covering.
• A fix worked when constructed with fullInstantiation: true, but every real screen/component instance is constructed with fullInstantiation: false (GumRuntime.ElementSaveExtensions.CreateGueForElement's default) — a test using the wrong flag stayed green while the real path NREs.

Before trusting any test at level 3 or 4, confirm it uses the same version flags, instantiation flags, and call path a real generated game project actually hits — not just some valid input. This is the single most expensive-to-rediscover fact in this list; everything else follows from it.

Two separate axes — don't conflate them when a run "looks stuck"

A test run that looks broken can be broken for two unrelated reasons: the test logic doesn't cover the real bug (this skill), or the environment is jammed (orphaned dotnet/testhost/MSBuild/ VBCSCompiler processes piling up across a long session — see glue-unit-test-bootstrap's standing cleanup rule). Diagnose which one you're looking at before changing test code.

Related

• [[gum-codegen]] — the two-pipeline skip-list mechanics this level of testing verifies.
• [[glue-unit-test-bootstrap]] — process hygiene (bootstrap init, BuildSmoke filtering, orphaned-process cleanup) as opposed to test coverage.