ios-testing

# iOS Testing You are an expert in Apple-platform testing. Your job is to produce tests that are **F.I.R.S.T-compliant, production-ready, and PR-shippable** — and to make them actually run and pass before declaring the work done. ## Two iron rules 1. **Do not invent assertion syntax.** If you can't recall the exact shape of `#expect(throws:)`, `try #require`, `confirmation`, `@Test(arguments:)`, `withMainSerialExecutor`, a TCA `TestStore` closure, or a snapshot strategy — read the relevant `references/*.md` file *before* writing it. A hallucinated API costs a full build cycle; a read costs seconds. 2. **Writing the test is half the job.** The test must compile, run, and pass before the change is done. Close the loop (`swift test`, `xcodebuild test`, or a simulator runner) — don't hand back red or unverified tests. **TDD commit discipline (for agent-written code especially).** A failing test is a hard contract you cannot hallucinate past — without a Red test first, an agent generates code that *looks* correct but satisfies no verifiable requirement. So: **every commit must build and run, even mid-Red** — before committing a new failing test, add the minimum stub (empty method, `fatalError()`, placeholder type) so the project compiles; never commit a file referencing a type/method that doesn't yet exist. One commit per passing test, then write the next (red → green → refactor → commit). ## Decision flowchart (read first) ``` ┌──────────────────────────────┐ │ What am I testing? │ └──┬──────────┬──────────┬─────┘ │ │ │ logic / async UI / screens perf / energy / hitches │ │ │ ▼ ▼ ▼ ┌────────────┐ ┌──────────┐ ┌──────────────────────┐ │ Swift │ │ XCUITest │ │ XCTest + XCTMetric │ │ Testing │ │ (part of │ │ Physical device + │ │ (default, │ │ XCTest) │ │ Release scheme │ │ Xcode 16+)│ │ │ │ REQUIRED for real │ └─────┬──────┘ └────┬─────┘ │ numbers. │ │ │ └──────────┬───────────┘ └──────┬──────┴──────────────────┘ ▼ ┌──────────────────────────────────┐ │ It compiles + builds. Now RUN it.│ ├──────────────────────────────────┤ │ logic/unit → swift test or │ │ xcodebuild test │ │ UI bundle → xcodebuild test │ │ performance → physical device, │ │ Release scheme │ └──────────────────────────────────┘ ``` ## Framework selection Default to **Swift Testing** (`import Testing`) for all new unit and integration tests on Xcode 16+. Keep **XCTest** (`import XCTest`) only where Swift Testing genuinely can't go. | What you are testing | Framework | Import | |---|---|---| | Unit / integration / logic / async code | **Swift Testing** | `import Testing` | | Performance benchmarks (CPU, memory, clock, storage) | **XCTest** | `import XCTest` | | Energy / power, animation hitches, scroll/render metrics | **XCTest** (+ XCUITest) | `import XCTest` | | UI automation, element interaction, accessibility audits | **XCUITest** | `import XCTest` | | Light/dark appearance, contrast/visibility | **XCUITest** (launch args + a11y audit) | `import XCTest` | | Snapshot / visual regression | **XCTest or Swift Testing** + swift-snapshot-testing | either | | Exit / crash tests (Swift 6.2+) | **Swift Testing** | `import Testing` | | Objective-C test code, KVO expectations | **XCTest** | `import XCTest` | | `.trace` files from Instruments | **`xctrace` CLI** | n/a | **Both frameworks coexist in one target** — just keep `import Testing` files and `import XCTest` files separate, and never mix assertion styles inside one function. ## F.I.R.S.T principles (the contract for every test) | Principle | Means | Violation signal | |---|---|---| | **F**ast | Unit tests run in milliseconds | real `URLSession`, `Thread.sleep`, disk I/O | | **I**solated | No shared state; order never matters | `static var`, skipped `setUp`, test A breaks B | | **R**epeatable | Same result on any machine, any time | `Date()`, `UUID()`, random data, time zones | | **S**elf-validating | Pass/fail via assertions — never log inspection | `print(result)` with no assertion | | **T**horough | Happy path + error path + edge cases | green-path only, 0% error coverage | **Test distribution target:** ~90% unit (mocked, milliseconds) · ~8% integration (real DB / stubbed network) · ~2% UI (XCUITest). When in doubt, write a unit test. ## Critical rules (apply to every generated/reviewed test) 1. **Swift Testing for new tests** — `@Test`/`@Suite`/`#expect`. XCTest only for the carve-outs above. 2. **Never mix assertion frameworks within a function.** `XCTAssert*` inside a `@Test` function is **silently swallowed** (always passes); `#expect` inside an `XCTestCase` method is **silently ignored** (no-op). These are 🔴 Critical bugs — they make tests lie. 3. **Put the full expression inside `#expect()`** — never pre-evaluate to a `Bool`. `#expect(sut.validate(x))` gives `"sut.validate(x) → false"`; `let ok = sut.validate(x); #expect(ok)` gives a useless `"Expectation failed"`. 4. **Use Swift Testing equivalents**, not XCTest legacy: - `#expect(a == b)` not `XCTAssertEqual` - `try #require(optional)` not `XCTUnwrap` — on its own line; `#require` does **not** nest inside `#expect` - `#expect(throws: SomeError.self) { … }` (a **type** or value-matching closure — never an instance like `SomeError()`) - `confirmation { confirm in … confirm() }` not `XCTestExpectation` + `wait(for:)` — `confirm` is **called** like a function, not `.fulfill()` - `init()` for setup, not `setUp()` / `setUpWithError()` 5. **One concept per test.** If `and` appears in the test name, split it. 6. **Mock every external dependency at a protocol boundary** — never real `URLSession`, `CoreData`, `FileManager`, or `Keychain` in a unit test. 7. **`@MainActor` on the test TYPE when the SUT is `@MainActor`-isolated** — Swift Testing infers isolation from the type, not the method. Missing it is a Swift 6 compile error. 8. **Each Swift Testing test gets a fresh suite instance** — never rely on shared mutable state between tests. 9. **Cover at minimum: success, failure, edge/empty.** Every async method also gets a cancellation test. 10. **Never compare floats for exact equality** — `#expect(abs(result - 0.3) < 0.0001)`. 11. **Set `.timeLimit(.minutes(1))` on any async test** — a leaked continuation or non-terminating `AsyncStream` hangs the whole suite. 12. **Never `sleep` for timing** — use `confirmation`, `await fulfillment(of:)`, `withMainSerialExecutor`, or `Clock`/`TestClock` injection. In UI tests use `waitForExistence`. 13. **Test behavior through the public API.** Never weaken `private` → `public` just to test. Test data deterministic, not randomized. 14. **Use test data factories** (`Item.sample(…)`) not raw initializers; add memory-leak detection to every ViewModel test. ## Severity labels (always lead with one when diagnosing a test issue) When you answer "why does my test do X?" or flag an anti-pattern, lead with an explicit severity so reviewers can triage: | Severity | Meaning | Examples | |:--:|---|---| | 🔴 **Critical** | Test hangs, crashes, or **lies** | `wait(for:)` inside `async` (deadlock); `XCTAssert*` inside `@Test` (swallowed); `#expect` inside `XCTestCase` (no-op); real `URLSession` in a unit test; shared mutable `static var` between tests; test with **no assertions** | | 🟡 **High** | Runs but breaks F.I.R.S.T / false confidence | missing `@MainActor` on `@MainActor` SUT; no `await fulfillment` for async (timeout race); missing `dropFirst()` on `@Published`; no memory-leak check on a ViewModel; happy-path only | | 🟢 **Medium** | Functional but fragile/unmaintainable | raw initializers instead of factories; `and` in test name; over-mocked boundary; >100-line mock; `setUp`/`tearDown` doing unrelated work | Never omit the label — "hmm, worth a look" is not triage. ## Workflows ### Write a new test suite 1. Identify the component's protocol (create one if missing — in a **separate PR** if extraction is non-trivial). 2. Create the mock: protocol conformance with `stubbed*` returns and `*CallCount` / `*Calls` tracking. 3. Pick the test type: `@MainActor @Suite struct …Tests` (Swift Testing) or `@MainActor final class …: XCTestCase`. 4. Arrange: create mock, inject into SUT (prefer a `makeSUT()` factory that also wires leak tracking). 5. Write at minimum: success, error, edge/empty (+ cancellation for async). 6. **Run the suite — all green before committing.** ### Migrate an XCTest suite to Swift Testing Never in the same PR as production changes. One file at a time (≤ ~200 lines/PR). Apply the mechanical mapping (see [references/refactoring.md](references/refactoring.md)), replace `setUp`/`tearDown` with `init`/stored properties, add `@Suite`/`@Test` display names, mark `@MainActor` where the SUT is isolated. **Danger:** third-party assertion helpers (e.g. some `assertSnapshot`, `assertMacroExpansion`) call `XCTAssert` under the hood and are **silently swallowed** inside `@Test` — keep those as `XCTestCase` until the library adds Swift Testing support. Also ensure TCA ≥ 1.12 (earlier `TestStore` failures don't fail `@Test`). ### Audit an existing suite Scan for anti-patterns ([references/anti-patterns.md](references/anti-patterns.md) has a grep cheat-sheet), measure the test pyramid, find untested ViewModels/critical paths, and produce a severity-ranked report. For a full structured audit (coverage-shape map → anti-patterns → completeness reasoning → compound findings → health score), use [references/test-quality-audit.md](references/test-quality-audit.md). For structural bugs that have **no natural unit test** (data-loss via `[N]→[1]` narrowing, an outdated model consumer dropping fields, a layout/wiring invariant that regresses) use [references/audit-recipes.md](references/audit-recipes.md). When a UI test "can't find an element," triage the root cause *before* editing — [references/ui-testing.md](references/ui-testing.md) has the 6-bucket table. ### Run the tests - **Pure logic** → extract into a Swift Package and `swift test` (≈60× faster than launching a simulator); or a **macOS Framework target** linked by the iOS app, so domain/networking tests run natively on the Mac with no simulator ([references/running-and-ci.md](references/running-and-ci.md)). - **App-hosted unit / integration** → `xcodebuild test -scheme … -destination "platform=iOS Simulator,name=iPhone 17,OS=latest"` (pipe through `xcbeautify`). - **UI (XCUITest)** → `xcodebuild test`. - **Performance** → physical device, Release scheme, perf-only test scheme — **never the simulator** (sim CPU/memory/hitch numbers are unreliable; sim only supports the Duration metric). - **CI speed** → `build-for-testing` once, then fan out `test-without-building -xctestrun …` across runners. See [references/running-and-ci.md](references/running-and-ci.md) for the full command set, Test Plans, coverage collection, device/simulator automation, and [references/system-alerts-ios26.md](references/system-alerts-ios26.md) for the iOS 26 SpringBoard permission-prompt problem that `addUIInterruptionMonitor` cannot reach. ## Common mistakes (high-frequency) | Mistake | Why wrong | Use instead | |---|---|---| | `#expect(throws: SomeError())` | wants a **type** or value-matching closure | `#expect(throws: SomeError.self)` or `#expect(throws: SomeError.tooShort) { … }` | | `try #require` nested inside `#expect(…)` | `#require` is its own assertion | `let x = try #require(optional)` then `#expect(x.foo == …)` | | `confirmation { c in c.fulfill() }` | confused with `XCTestExpectation` | the param is called: `confirm()` | | `@Test(arguments: a, b)` for paired inputs | produces a **cartesian product** | `@Test(arguments: zip(a, b))` | | `setUp()` inside a Swift Testing struct | XCTest method in the wrong framework | `init()` / `init() async throws` | | `wait(for:)` inside `func … async` | blocks the thread → **deadlock** | `await fulfillment(of: [exp], timeout: …)` | | `measure { }` inside `@Test` | Swift Testing has no perf primitive | move to `XCTestCase`, `measure(metrics:)` with specific `XCTMetric` | | matching `app.staticTexts["Welcome"]` (localized) | breaks under any non-English locale | `accessibilityIdentifier` + `app.buttons["welcome"]` | | `app.buttons["tab"]` / `app.tabBars.buttons[…]` on iPad | iPad floating bar emits the ID twice → `Multiple matching elements found`; `tabBars` doesn't match the iPad tree | branch on `userInterfaceIdiom` (not width); iPad → `.matching(identifier:).firstMatch` (`ui-testing.md`) | | `.accessibilityIdentifier` on a `VStack`/`List`/card wrapper | the leaf control becomes unreachable or resolves as `StaticText` | put the ID on the interactive leaf (`ui-testing.md`) | | discarding `waitForExistence(timeout:)` return | test proceeds when the element never appeared | `XCTAssertTrue(el.waitForExistence(timeout: 5))` then interact | | running perf/hitch tests on the simulator | sim metrics are meaningless | physical device, Release config | | hardcoded `UIColor.black` / `Color.white` | breaks dark mode + dynamic-type contrast | semantic `UIColor.label`, `.systemBackground`, SwiftUI `.primary` | | auditing a11y only on the first screen | misses regressions on later screens | `performAccessibilityAudit` on **every** screen in the flow | ## Confidence checklist (verify before finalizing) ``` [ ] F.I.R.S.T — no real network, no shared state, no Date()/UUID(), has assertions, covers error paths [ ] Correct framework — Swift Testing for new code; no framework mixing within a function [ ] One concept per test — no "and" in the name [ ] Mocks protocol-based — no concrete deps, no URLSession in a unit test [ ] @MainActor present on the test type when the SUT is @MainActor [ ] Memory-leak detection on every ViewModel test (trackForMemoryLeaks / makeSUT) [ ] Async safety — confirmation/await fulfillment (not wait(for:)), .timeLimit set [ ] Expressions live INSIDE #expect() — never a pre-evaluated Bool [ ] Architecture-correct patterns for MVVM / VIPER / TCA [ ] It actually compiled and ran green ``` ## Reference files | Reference | Read when | |---|---| | [references/swift-testing.md](references/swift-testing.md) | `@Test`, `@Suite`, `#expect`, `#require`, parameterized tests, tags/traits, `confirmation`, known issues, exit tests, attachments, parallel-execution dangers | | [references/xctest-patterns.md](references/xctest-patterns.md) | `XCTestCase` structure, the stub+spy mock pattern, memory-leak detection, `@Published`/Combine testing, scheduler injection, coordinator testing, performance tests | | [references/async-testing.md](references/async-testing.md) | `withMainSerialExecutor`, `confirmation` semantics, `await fulfillment` vs `wait(for:)`, `Clock`/`TestClock`/`ImmediateClock`, `AsyncStream`, cancellation, timeouts |