gum-codegen-test-rigor

# Gum Codegen: What Actually Proves a Fix Works Glue's Gum codegen (`gum-codegen` skill) emits C# source text with no compile-time link to the real runtime types it targets. That's why weak verification repeatedly looks green while real bugs ship — this skill is the record of that pattern recurring, so the next fix doesn't restart at the bottom of this list. ## Verification strength ladder — each level misses a different bug class From weakest to strongest, all four were tried in sequence on the same fix before it actually worked: 1. **String-matching generated source** (`generatedSource.ShouldContain/ShouldNotContain(...)`). Proves text presence only. Misses everything about correctness. 2. **Reflection-based member-contract checks** (`GumRuntimeMemberContractTests`). Proves a *named member* exists on the *contained* type, but only for patterns the check explicitly matches (e.g. a `ContainedXxx.Member` regex). Missed a base-class (`GraphicalUiElement.RenderableComponent`) assignment entirely — wrong access shape, so the pattern never matched it, so it never looked. 3. **Real compile against real assemblies** (BuildSmoke `*CreationSmokeTests` / `GumGeneratedCodeCompilesTests`). Proves the generated code *compiles*. Says nothing about runtime behavior — code that compiles clean can still NRE the instant its constructor runs. 4. **Real instantiate-and-execute.** Actually construct the generated runtime the way production code does, and run it. This is the only level that catches a construction-time `NullReferenceException`. Nothing weaker than this can. If a fix "passed" at one level and still broke in the field, the next debugging step is always "move up this ladder," not "add another test at the same level." ## The sharper landmine: matching the real call path, not just reaching a level Being at the top of the ladder isn't sufficient either — a test can compile-and-run and *still* miss the bug if it doesn't reproduce the exact parameters/state production code actually uses: - A BuildSmoke sweep silently defaulted `GumProjectSave.Version`, so it always exercised the v2 branch and never touched the v3 code path it was supposedly covering. - A fix worked when constructed with `fullInstantiation: true`, but every real screen/component instance is constructed with `fullInstantiation: false` (`GumRuntime.ElementSaveExtensions.CreateGueForElement`'s default) — a test using the wrong flag stayed green while the real path NREs. **Before trusting any test at level 3 or 4, confirm it uses the same version flags, instantiation flags, and call path a real generated game project actually hits — not just *some* valid input.** This is the single most expensive-to-rediscover fact in this list; everything else follows from it. ## Two separate axes — don't conflate them when a run "looks stuck" A test run that looks broken can be broken for two unrelated reasons: the test logic doesn't cover the real bug (this skill), or the *environment* is jammed (orphaned `dotnet`/`testhost`/`MSBuild`/ `VBCSCompiler` processes piling up across a long session — see `glue-unit-test-bootstrap`'s standing cleanup rule). Diagnose which one you're looking at before changing test code. ## Related - [[gum-codegen]] — the two-pipeline skip-list mechanics this level of testing verifies. - [[glue-unit-test-bootstrap]] — process hygiene (bootstrap init, BuildSmoke filtering, orphaned-process cleanup) as opposed to test *coverage*.