# Gum Codegen Test Rigor

> Gum Codegen: What Actually Proves a Fix Works

- Skill: `vchelaru/gum-codegen-test-rigor` (Agent Skill)
- Install (CLI): `npx skillmds@latest add vchelaru/gum-codegen-test-rigor`
- Raw SKILL.md: https://api.skillmd.com/api/skills/vchelaru/gum-codegen-test-rigor/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: vchelaru (https://skillmd.com/u/vchelaru)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/vchelaru/gum-codegen-test-rigor

---


# Gum Codegen: What Actually Proves a Fix Works

Glue's Gum codegen (`gum-codegen` skill) emits C# source text with no compile-time link to the real
runtime types it targets. That's why weak verification repeatedly looks green while real bugs ship — this
skill is the record of that pattern recurring, so the next fix doesn't restart at the bottom of this list.

## Verification strength ladder — each level misses a different bug class

From weakest to strongest, all four were tried in sequence on the same fix before it actually worked:

1. **String-matching generated source** (`generatedSource.ShouldContain/ShouldNotContain(...)`). Proves
   text presence only. Misses everything about correctness.
2. **Reflection-based member-contract checks** (`GumRuntimeMemberContractTests`). Proves a *named member*
   exists on the *contained* type, but only for patterns the check explicitly matches (e.g. a
   `ContainedXxx.Member` regex). Missed a base-class (`GraphicalUiElement.RenderableComponent`) assignment
   entirely — wrong access shape, so the pattern never matched it, so it never looked.
3. **Real compile against real assemblies** (BuildSmoke `*CreationSmokeTests` /
   `GumGeneratedCodeCompilesTests`). Proves the generated code *compiles*. Says nothing about runtime
   behavior — code that compiles clean can still NRE the instant its constructor runs.
4. **Real instantiate-and-execute.** Actually construct the generated runtime the way production code
   does, and run it. This is the only level that catches a construction-time `NullReferenceException`.
   Nothing weaker than this can.

If a fix "passed" at one level and still broke in the field, the next debugging step is always "move up
this ladder," not "add another test at the same level."

## The sharper landmine: matching the real call path, not just reaching a level

Being at the top of the ladder isn't sufficient either — a test can compile-and-run and *still* miss the
bug if it doesn't reproduce the exact parameters/state production code actually uses:

- A BuildSmoke sweep silently defaulted `GumProjectSave.Version`, so it always exercised the v2 branch and
  never touched the v3 code path it was supposedly covering.
- A fix worked when constructed with `fullInstantiation: true`, but every real screen/component instance
  is constructed with `fullInstantiation: false` (`GumRuntime.ElementSaveExtensions.CreateGueForElement`'s
  default) — a test using the wrong flag stayed green while the real path NREs.

**Before trusting any test at level 3 or 4, confirm it uses the same version flags, instantiation flags,
and call path a real generated game project actually hits — not just *some* valid input.** This is the
single most expensive-to-rediscover fact in this list; everything else follows from it.

## Two separate axes — don't conflate them when a run "looks stuck"

A test run that looks broken can be broken for two unrelated reasons: the test logic doesn't cover the
real bug (this skill), or the *environment* is jammed (orphaned `dotnet`/`testhost`/`MSBuild`/
`VBCSCompiler` processes piling up across a long session — see `glue-unit-test-bootstrap`'s standing
cleanup rule). Diagnose which one you're looking at before changing test code.

## Related
- [[gum-codegen]] — the two-pipeline skip-list mechanics this level of testing verifies.
- [[glue-unit-test-bootstrap]] — process hygiene (bootstrap init, BuildSmoke filtering, orphaned-process
  cleanup) as opposed to test *coverage*.