name: gum-codegen-test-rigor
description: What actually proves a Gum-codegen fix works, vs. what only looks like it does. Triggers: "tests pass but it's still broken", CS1061/CS0200/NRE surviving a green suite, verifying StandardsCodeGenerator/StateCodeGenerator/GueDerivingClassCodeGenerator changes.
Gum Codegen: What Actually Proves a Fix Works
Glue's Gum codegen (gum-codegen skill) emits C# source text with no compile-time link to the real
runtime types it targets. That's why weak verification repeatedly looks green while real bugs ship — this
skill is the record of that pattern recurring, so the next fix doesn't restart at the bottom of this list.
Verification strength ladder — each level misses a different bug class
From weakest to strongest, all four were tried in sequence on the same fix before it actually worked:
- String-matching generated source (
generatedSource.ShouldContain/ShouldNotContain(...)). Proves
text presence only. Misses everything about correctness.
- Reflection-based member-contract checks (
GumRuntimeMemberContractTests). Proves a named member
exists on the contained type, but only for patterns the check explicitly matches (e.g. a
ContainedXxx.Member regex). Missed a base-class (GraphicalUiElement.RenderableComponent) assignment
entirely — wrong access shape, so the pattern never matched it, so it never looked.
- Real compile against real assemblies (BuildSmoke
*CreationSmokeTests /
GumGeneratedCodeCompilesTests). Proves the generated code compiles. Says nothing about runtime
behavior — code that compiles clean can still NRE the instant its constructor runs.
- Real instantiate-and-execute. Actually construct the generated runtime the way production code
does, and run it. This is the only level that catches a construction-time
NullReferenceException.
Nothing weaker than this can.
If a fix "passed" at one level and still broke in the field, the next debugging step is always "move up
this ladder," not "add another test at the same level."
The sharper landmine: matching the real call path, not just reaching a level
Being at the top of the ladder isn't sufficient either — a test can compile-and-run and still miss the
bug if it doesn't reproduce the exact parameters/state production code actually uses:
- A BuildSmoke sweep silently defaulted
GumProjectSave.Version, so it always exercised the v2 branch and
never touched the v3 code path it was supposedly covering.
- A fix worked when constructed with
fullInstantiation: true, but every real screen/component instance
is constructed with fullInstantiation: false (GumRuntime.ElementSaveExtensions.CreateGueForElement's
default) — a test using the wrong flag stayed green while the real path NREs.
Before trusting any test at level 3 or 4, confirm it uses the same version flags, instantiation flags,
and call path a real generated game project actually hits — not just some valid input. This is the
single most expensive-to-rediscover fact in this list; everything else follows from it.
Two separate axes — don't conflate them when a run "looks stuck"
A test run that looks broken can be broken for two unrelated reasons: the test logic doesn't cover the
real bug (this skill), or the environment is jammed (orphaned dotnet/testhost/MSBuild/
VBCSCompiler processes piling up across a long session — see glue-unit-test-bootstrap's standing
cleanup rule). Diagnose which one you're looking at before changing test code.
Related
- [[gum-codegen]] — the two-pipeline skip-list mechanics this level of testing verifies.
- [[glue-unit-test-bootstrap]] — process hygiene (bootstrap init, BuildSmoke filtering, orphaned-process
cleanup) as opposed to test coverage.
