| name | edmund-validation-and-qa |
| description | What counts as EVIDENCE in the Edmund Markdown editor, and how to add tests. Load when writing or adjusting tests, deciding what proof a fix needs before it can ship, judging whether a change is actually "verified", interpreting the test suite, or adding a golden/regression check. Covers the evidence hierarchy (headless is necessary but not sufficient for live-input bugs), the test-suite map, the Swift Testing helpers, how to add a test, the golden inventory, visual QA, and performance evidence. Not the gating rules themselves (edmund-change-control) or how to run the live repro (edmund-live-repro-and-diagnostics).
|
Edmund validation & QA
The discipline of proof. The central lesson: a green headless test is
necessary but not sufficient for the bug classes that cost the most time here.
Know which evidence each change actually requires.
Framework: Swift Testing (import Testing, @Test, #expect, #require)
— NOT XCTest. Verified 2026-07-05: 810 @Test cases across Tests/EdmundTests/.
1. The evidence hierarchy
Ordered weakest → strongest. The rule at the bottom names which class each
change type requires.
| Class | What it proves | Blind spot |
|---|
(a) Headless unit test (makeEditor()) | model/parse/style/undo logic | Cannot exercise deferred AppKit machinery — runs it synchronously |
(b) Windowed unit test (real NSWindow + NSScrollView, real deleteBackward) | + layout, viewport, first responder | still not real event-loop pacing / IME / drag |
| (c) Frozen live ReproScript repro (exact script + document) | the live input/timing mechanism | needs a debug build + ~1 min/run |
(d) Soak script green across 4–5 cycles, byte-identical final rawLen | armed-state + determinism | slow |
| (e) Screencapture pixel measurement | anything that DRAWS | manual |
The trap that defines this skill: the delete-drift round-6 regression test
passes with AND without the fix — the harness runs the queued selection fixup
synchronously, so headless can't see the bug. For caret/IME/drag/viewport-timing
bugs, class (a) is not evidence of a fix; you need (c)+(d).
Required evidence by change type (gates enforced in edmund-change-control):
| Change | Requires |
|---|
| Pure logic (parse/style/model) | (a) |
| Viewport/lazy-layout | (b), often (e) |
| Anything that draws | (a where testable) + (e) in light AND dark |
| Edit-pipeline / selection / IME / undo | (a) to lock the headless contract + (c) frozen repro + (d) soak |
| Release | see edmund-release-and-operate |
2. Test-suite anatomy
Run: swift test (full suite; ARCHITECTURE cites ~750+ tests ≈10s — 810 @Test
cases as of 2026-07-05). One suite: swift test --filter <Suite>. swift test
also runs automatically as a Stop hook (.claude/settings.json) at the end of
any turn touching code, so failures surface before you commit.
Map of Tests/EdmundTests/ (what the files actually cover):
- Parsing:
BlockParserTests, SyntaxHighlighterTests, IncrementalParseFuzzTests, PendingEditTests, EscapeRenderingTests, HTMLTagRenderingTests, EmojiRenderingTests, LineEndingTests.
- Rendering / styling:
BlockStylingTests, CalloutRenderingTests, CalloutTests, CommentRenderingTests, NestedBlockStylingTests, InlineStylingTests, MathRenderingTests, ImageRenderingTests, CodeHighlighterTests, FootnoteTests, WikiLinkTests, TableAlignmentTests, RenderingRegressionTests.
- Editor behavior:
EditorIndentationTests, ListContinuationTests, BlockquoteContinuationTests, BlockquoteDeletionTests, FormattingTests, ActiveBulletMarkerTests, NewListItemAlignmentTests.
- Edit-pipeline integrity (the costly class):
BypassedEditSyncTests, MarkedTextDesyncTests, WrappedParagraphCaretTests, EditorDiagnosticsTests, UnmatchedDebugTests, InternationalInputTests.
- Viewport / layout / undo:
LazyRenderingTests, ScrollStabilityTests, HeightStabilityTests, TypewriterCenteringTests, UndoRedoViewportTests, EditorUndoTests, RecomposeTests, RecomposeEquivalenceTests.
- Export / read mode:
HTMLRendererTests, DocumentHTMLTests, ReadModeWebViewTests, HTMLThemeTests, EditorThemeTests, ViewModeTests, ContentWidthTests.
- Infra / harness:
LogTests, CrashReporterTests, PerfHarnessTests, StatusBarPrefsTests, FileIntegrationTests, EditorDocumentTests, TestHelpers.swift. _RenderDump.swift / _RenderEdit.swift are local dev tools (gitignored intent) that dump Read-mode HTML / edit output for tmp/sample.md — not part of the assertion suite.
Two invariant-guarding patterns worth copying:
- Recompose equivalence (
RecomposeEquivalenceTests, helper
assertMatchesFullRecomposeOracle): an incremental restyle must produce the
same result as a full recompose. This is the safety net for the lazy /
incremental styling paths.
- Incremental-parse fuzz (
IncrementalParseFuzzTests): random edits must
keep the parser's window-reparse consistent with a full reparse.
3. Test helpers (TestHelpers.swift)
@MainActor helpers you build on (verified exports):
| Helper | Use |
|---|
makeEditor() | an EditorTextView on the real TK2 chain (mirrors Document.makeWindowControllers) |
ensureFullLayout(...) | force layout so geometry is real, not estimated |
type(...), paste(...), pressEnter(), pressBackspace() | drive edits |
activateBlock(...) | move the caret / active block |
displayText(...), attrs(...), font(...), fgColor(...) | inspect styled output |
isHidden / isInvisible / isDimmed | assert delimiter hiding |
blockDecoration(...), textBlockDifference(...) | inspect decorations / catch TK1-reverting table attrs |
expectedFullComposition(...), drainAllStyling(), assertMatchesFullRecomposeOracle(...) | equivalence-oracle checks |
makeLargeMarkdown(...), sentence(...) | build big fixtures for perf/viewport |
styleBlock(_:cursorPosition:) is a method on EditorTextView
(Rendering/EditorTextView+Rendering.swift), called from tests — it renders one
block to an attributed string.
4. How to add a test
Skeleton (Swift Testing, @MainActor because the editor is main-thread):
import Testing
import AppKit
@testable import EdmundCore
@MainActor
@Test func deletingAtCalloutBottomKeepsInvariant() {
let editor = makeEditor()
editor.loadRawSource("> [!note]\n> body\n")
drainAllStyling()
#expect(editor.rawSource == editor.string)
}
Rules:
- Every bug fix ships with a test even if it can't discriminate a live-only
mechanism — it still locks the headless contract so a future refactor
can't silently re-break the model half. For the live half, keep the frozen
.repro script alongside (edmund-live-repro-and-diagnostics).
- Assert the invariant where you can (
rawSource == string), not just the
surface symptom — that's what BypassedEditSyncTests / MarkedTextDesyncTests
do.
- Name the test for the behavior/bug, put edit-pipeline repros next to their
siblings (the
*Desync* / *Bypassed* / *CaretTests families).
- New drawing behavior additionally needs a screencapture check (§6).
5. Golden / certified inventory
RenderingRegressionTests + RecomposeEquivalenceTests are the closest
thing to golden checks — they pin styled output and incremental-vs-full
equivalence.test-files/*.md (callout.md, decorations.md, math.md, menu.md,
test.md) are the user's manual test corpus — hand-testing fodder. Do
not rewrite them to fit an automated test; build your own fixtures (helpers
in §3, or a scratch dir).misc/bug-repros/ holds field evidence (.mov screen recordings + .log
traces) for open bugs — reference these when reproducing, don't delete them.
6. Visual QA
Anything that draws is verified by screencapture pixel measurement, in
light AND dark mode (per misc/before-you-release.md), never by eyeballing
headless layout. Method + scripts: edmund-live-repro-and-diagnostics §5.
Headless layout tests (HeightStabilityTests, ScrollStabilityTests) check
geometry numbers but cannot confirm the pixels are right.
7. Flakiness & CI
- A
fix/flaky-math-test branch exists in history — math rendering has shown
timing flakiness; if a math test flakes, check that branch's approach before
inventing a new one (verify: git log --oneline --all -- '*Math*').
- CI:
.github/workflows/ci.yml on macos-14, latest-stable Xcode, SPM cache
keyed on Package.resolved, concurrency: cancel-in-progress (private-repo
macOS minutes bill 10×). CI runs the same swift test.
8. Performance evidence
PerfHarnessTests measures the hot paths; makeLargeMarkdown builds big
fixtures. Use it (don't hand-time) for any perf claim.- The README claim "handles ~1–2MB files" and
fullLayoutMaxLength = 100_000
UTF-16 (EditorTextView.swift:80) mark the boundary between the full-layout
regime (≤100k, geometry is real) and the estimate regime (>100k, viewport
glitches possible). A perf/viewport claim must state which regime it was
measured in.
When NOT to use this skill
- The gate/branch/commit rules → edmund-change-control.
- Running the live repro or measuring pixels → edmund-live-repro-and-diagnostics.
- The caret-integrity investigation end to end → edmund-caret-integrity-campaign.
- Why a mechanism works → textkit2-appkit-reference / edmund-architecture-contract.
- Release verification → edmund-release-and-operate.
Provenance and maintenance
Verified 2026-07-05.
grep -rh '@Test' Tests/EdmundTests/*.swift | wc -l
grep -c 'import Testing' Tests/EdmundTests/TestHelpers.swift
grep -oE 'func [a-zA-Z]+' Tests/EdmundTests/TestHelpers.swift
ls Tests/EdmundTests/
grep -n 'fullLayoutMaxLength' Sources/EdmundCore/TextView/EditorTextView.swift
If a helper in §3 no longer greps it was renamed; update §3 and any test
skeletons. Re-derive the suite map from ls if files are added/removed.
