apple-integration-testing

name

apple-integration-testing

description

Use when writing integration tests for Apple platforms (Swift / XCTest) covering service layers, repositories, parsers, data pipelines, or any code whose correctness is verifiable through observable state (file contents, returned values, DB rows, exit codes) without launching the UI. Use when consolidating small unit tests into scenario-based integration tests, when a test passes but the feature is broken (vacuous assertion), or when an existing test silently skips a criterion.

Apple Integration Testing

Assertion-based integration tests for Apple platforms. Real dependencies, real files, real frameworks. One scenario test per pipeline, step-by-step assertions, no mocks of the thing under test.

This skill covers observable-state verification only — file contents, returned values, DB rows, exit codes. UI verification (layout, toasts, modals) is out of scope and belongs to a UI-driving skill.

When to Use

• Service layer, repositories, parsers, value logic, data pipelines on iOS / macOS
• Multi-stage pipelines (A → B → C → D) where a silent failure between stages is the bug class you fear
• External-dependency code: C/FFI, vendored libs, model loading, ScreenCaptureKit, AVFoundation, CoreData, OpenAPI repositories
• File I/O chains: input file → processing → output file (existence ≠ correctness)
• Consolidating many small unit tests into one integrated scenario per pipeline

When NOT to Use

• UI-visible criteria (layout, toasts, modals, focus, dialogs) — those need a UI driver, not XCTest
• Pure isolated value-object logic where one tiny unit test is genuinely sufficient
• Code that can't be instantiated without launching the full app — refactor first (extract logic, inject dependencies), then test

Core Principle

Test the pipeline, not the methods. One integrated test that exercises A → B → C → D catches real interaction bugs that isolated tests miss. When it fails, step-by-step assertions tell you exactly WHERE.

func test_fullScenario_featureName() throws {
    // STEP 1: Setup — create real test data
    let input = try makeRealInput()
    XCTAssertGreaterThan(input.count, 0, "Step 1: setup produced empty input")

    // STEP 2: Component A processes input
    let a = ServiceA().process(input)
    XCTAssertFalse(a.isEmpty, "Step 2: ServiceA produced empty output for non-empty input")

    // STEP 3: Component B receives A's output
    let b = ServiceB().consume(a)
    XCTAssertTrue(b.contains(expectedToken), "Step 3: ServiceB output missing expected token \(expectedToken)")

    // STEP 4: End-to-end verification — parse & validate content
    let final = try Parser.parse(b)
    XCTAssertEqual(final.kind, .success, "Step 4: final parse did not yield success")
}

Rules: One test per pipeline, not per method. Each step asserts with a unique failure message. Fail fast — don't run Step 3 if Step 2 failed. Real dependencies, no mocks. Clean up in tearDown. Remove redundant small tests already covered by the scenario. Keep small tests ONLY for error paths, boundary conditions, and pure logic not exercised by scenarios.

Integration Test Contract

Before writing tests, write a contract — even a short one in your PR description. It forces you to name what each test must prove.

# Integration Test Contract: <feature>

## Pipeline
A (input parser) → B (transform) → C (encoder) → D (output writer)

## Silent failure risks
- B may drop fields when input contains nested arrays
- C may produce a non-empty file with corrupt header bytes

## SCENARIO 1: full pipeline, happy path
- SETUP: temp dir, fixture file `input.json`
- ACTION: run pipeline end-to-end
- ASSERT_TYPE: behavioral
- ASSERT_CONTAINS: output file exists, size > 0, header == "MAGIC", parsed AST has N nodes
- FAIL_IF_BLOCKED: "FAIL: cannot verify SCENARIO 1 — fixture input.json missing"

## EDGE 1: input missing required field
- SCOPE: error path only
- ASSERT: throws ParseError.missingField("name")

Assertion types:

• behavioral — the mandatory before/after pattern (capture state before, perform action, capture after, assert state CHANGED, assert new state CONTAINS expected content). A change check alone passes for errors too. A content check alone doesn't prove the action caused the change. Both are required.
• state — verify a property matches an expected value (disabled, checked, count == N).
• existence — element is present. Only acceptable for "is the thing visible" criteria. Forbidden as a substitute for behavioral.

Behavioral Assertion Pattern (Swift / XCTest)

let before = service.currentState()
try service.performAction(input)
let after = service.currentState()

XCTAssertNotEqual(before, after, "AC2: state must change after performAction")
XCTAssertTrue(after.contains(expectedToken),
              "AC2: new state must contain \(expectedToken), got \(after)")

Both lines required for behavioral. One alone is vacuous.

Forbidden Guard Patterns (Swift)

Every contract criterion must either assert (success) or fail (blocked) — never silently skipped.

FORBIDDEN — silently skips when condition is false:

guard let result = action() else { return }              // assertion never runs
if let dir = findDirectory() { XCTAssertTrue(...) }      // skipped on nil

ALLOWED — fails loudly when prerequisite missing:

guard let result = action() else {
    XCTFail("action() returned nil — prerequisite for AC3 not met")
    return
}
let dir = findDirectory()
XCTAssertFalse(dir.isEmpty, "findDirectory() must return a non-empty path")

If a prerequisite cannot be established, FAIL explicitly with a FAIL_IF_BLOCKED message verbatim from the contract. Never paraphrase.

Single-Flow State Machine

When the contract has Phases that depend on each other, implement them in a single test function that flows Phase 1 → Phase 2 → … → Phase N.

Why: Splitting Phases into separate functions means each function starts from scratch and loses state Phase 3 needs from Phase 1.

Exceptions — separate test functions are allowed only for:

• A criterion requiring process relaunch (persistence across restart)
• A criterion requiring contradictory preconditions (e.g., feature OFF after feature ON)

Each exception function must re-establish its own preconditions from scratch.

Bypass Flag Ban

Flags that bypass real processing are banned in tests:

-generateTestTranscript
-useTestDownloads
-useFakeData

Only state config flags are allowed (e.g., -hasCompletedSetup YES to skip an onboarding wizard). State flags configure the app; they do not replace the code under test.

Equivalents to scan for: Simulated*, Fake*, Mock* in production targets — see "Never Simulate" below.

Real Test Content

Generate real inputs with system tools, not hardcoded canned outputs.

Feature	How
Audio processing	say "test content" -o /tmp/test.aiff then convert if needed
Transcription	say known text → feed to service → assert output contains known text
Window enumeration	real ScreenCaptureKit query of the running test process
Persistence	real CoreData store on disk in a temp dir; tear down after
Network repositories	real local server fixture or recorded responses replayed by URLProtocol — never a mock of the repository

Small but real: 2-second audio clips, minimal valid files, tiny models. Each test under 30s where possible (full-pipeline tests may be longer). Use temp directories and clean up in tearDown.

Never Simulate

Never put Simulated{Feature}Repository, Fake{Feature}, or Mock{Feature} in production code. Every service must use real framework APIs:

• Window enumeration → ScreenCaptureKit (not a hardcoded window list)
• Model inference → real ML framework (not Thread.sleep() + canned output)
• Media playback → AVPlayer (not a static image)

If a real API requires permissions or hardware that blocks progress, treat it as a blocker to escalate, not to stub. The only acceptable test doubles live in test targets — never in the running app.

Adding New Test Files (XcodeGen)

When adding a new .swift file to a test target, run xcodegen generate to regenerate .xcodeproj. The sources: directive in project.yml auto-discovers .swift files in the directory, but only after regeneration.

Never edit .pbxproj manually. Never use the Xcode GUI to change Build Settings on an XcodeGen-managed project — xcodegen generate overwrites those edits.

Running Tests

Run only the tests related to the current change — specific test class or file. Do NOT run the full test suite while iterating.

xcodebuild test \
  -project MyApp.xcodeproj \
  -scheme MyApp \
  -destination 'platform=macOS' \
  -only-testing:MyAppTests/PipelineIntegrationTests

Split build and test commands when the platform supports it so build errors surface immediately.

If output is too verbose, spawn a sub-agent (or pipe to a file you read separately) — do not pipe xcodebuild through | tail / | grep while the build is running, that hides failures.

Compliance Scans (run before declaring done)

Run these greps against the test files and the production code they exercise. Any non-CLEAN result needs a fix before merge.

# 1. Bypass flags in tests
grep -rn "generateTestTranscript\|useTestDownloads\|useFakeData" . --include="*.swift" || echo "CLEAN"

# 2. Stub functions in production (return "" / return [] as the only path)
grep -rn 'return ""$\|return \[\]$' . --include="*.swift" \
  | grep -v "Tests\|guard\|else\|catch\|//" || echo "CLEAN"

# 3. Test data generators in production code
grep -rn "testSentences\|generateTest\|hardcodedSegments" . --include="*.swift" \
  | grep -v "Tests" || echo "CLEAN"

# 4. Vacuous assertions (OR'd booleans always pass)
grep -rn "XCTAssertTrue.*||" . --include="*.swift" | grep "Tests" || echo "CLEAN"

# 5. Silent-skip guards in test files (if-let around assertions, no XCTFail)
TEST_FILE=path/to/IntegrationTests.swift
grep -n 'if let.*= .*{' "$TEST_FILE" \
  | grep -v "// optional\|cleanup\|Cleanup\|delete\|Delete" || echo "CLEAN"

# 6. Empty output artifacts (adjust paths/sizes per feature)
find ~/<AppName> -name "audio.m4a" -size -1k 2>/dev/null
find ~/<AppName> -name "transcript.jsonl" -empty 2>/dev/null
find ~/<AppName> -name "video.mp4" -size -10k 2>/dev/null

For every assertion in the test file, ask the emptied-handler test: if I deleted the body of the function under test, would this assertion still pass? If yes, the assertion is vacuous — strengthen it.

Quick Reference

Symptom	Cause	Fix
Test passes but feature is broken	Vacuous assertion (existence-only or change-only)	Apply behavioral pattern: assert change AND content
guard let around the assertion, test "passes" with no output	Silent-skip guard	Replace else { return } with else { XCTFail(...); return }
File exists at expected path → test green; user sees garbage	Existence check, no content validation	Parse the file; assert size, header, structure
Phase 3 fails with "no state" when run with all tests	Phases split across separate test functions	Combine into one test that runs Phase 1 → 2 → 3 in order
Test green; manual run shows broken pipeline	Mock somewhere in the chain	Remove mocks for the components under test; use real framework
Test compiles, won't run after adding new file	Forgot xcodegen generate	Run it; clean DerivedData if stale
Build setting reverted next run	Edited .xcodeproj directly under XcodeGen	Move setting into project.yml, regenerate

Common Mistakes

Mistake	Fix
Using XCTAssertNotEqual(before, after) alone for behavioral	Add XCTAssertTrue(after.contains(expected)) — both required
if let result = service.run() { XCTAssertTrue(result.ok) }	guard let result = ... else { XCTFail(...); return }
Many small test_* per service, none exercising the pipeline	One scenario test exercising A → B → C → D with step-by-step asserts
Mocking the repository the test is meant to verify	Use a real on-disk store / real local server in temp dir
Hardcoded fixture audio without speech content	say "known phrase" -o tmp.aiff, assert transcript contains "known phrase"
Thread.sleep(forTimeInterval: 3) waiting for async work	Use XCTestExpectation or await — no fixed sleeps
Test file added to repo, ignored by Xcode	Run xcodegen generate, re-clean DerivedData
Running entire suite on every change	Run only -only-testing:Module/ClassName while iterating

Adopting in a New Project

1. Decide which pipelines need integration coverage (the data flows where silent failures hurt most).
2. Write the integration test contract for one pipeline (pipeline name, silent failure risks, scenarios with ASSERT_CONTAINS).
3. Implement one scenario test that follows the contract literally — step assertions, behavioral pattern, real dependencies.
4. Run the compliance scans above; clean every flag.
5. Delete the redundant small unit tests the scenario now covers.
6. Repeat per pipeline.