example-testing-setup

name	example-testing-setup
description	Setup guide for testing code examples in documentation CI — covers doctest, snippet extraction, execution sandboxing, and the failure-handling policy to ensure every published example actually runs.

Example Testing Setup

When to Use This

Your docs contain code examples that may silently rot as the API evolves. Use this skill to wire example testing into CI so every example is verified on every PR and release.

Strategy Selection

Doc toolchain	Example type	Recommended approach
Any	Shell/CLI commands	Extract + run in CI with script -q -c "..." or bash
Python project	Python snippets	doctest or pytest with extracted snippets
Node/JS project	JS/TS snippets	Jest or Vitest with imported snippet files
OpenAPI spec	Request/response examples	Dredd or Schemathesis against a running service
Any	Multi-language	mdbook's {{#include}} or Docusaurus @theme/CodeBlock with test runner

Core principle: examples live in source files that are imported into docs, never pasted inline. The test runs the source file; the doc renders it.

Pattern 1 — Include-from-File

Instead of inline fenced code blocks, maintain runnable source files:

docs/
  examples/
    quickstart.py       ← the actual runnable file
    auth-example.py
  quickstart.md         ← imports from examples/

In Docusaurus MDX:

import CodeBlock from '@theme/CodeBlock';
import QuickstartSource from '!!raw-loader!./examples/quickstart.py';

<CodeBlock language="python">{QuickstartSource}</CodeBlock>

The test CI step simply runs all files under docs/examples/:

- name: Test documentation examples
  run: |
    for f in docs/examples/*.py; do
      echo "Testing $f"
      python "$f" || exit 1
    done

Pattern 2 — Doctest (Python)

Embed testable examples directly in docstrings and markdown using doctest:

def calculate_discount(price: float, pct: float) -> float:
    """
    >>> calculate_discount(100.0, 10)
    90.0
    >>> calculate_discount(50.0, 0)
    50.0
    """
    return price * (1 - pct / 100)

CI step:

- name: Run doctests
  run: python -m doctest -v docs/**/*.md

Limit doctest to self-contained expressions. Long setup sequences belong in Pattern 1.

Pattern 3 — OpenAPI Request/Response Validation

For API reference docs, validate that request/response examples in the OpenAPI spec actually match the live service:

# Using Schemathesis
- name: Test API examples
  run: |
    schemathesis run docs/openapi.yaml \
      --base-url http://localhost:8080 \
      --checks all \
      --contrib-openapi-formats-uuid \
      --hypothesis-max-examples 50

Or with Dredd for exact example matching:

- name: Dredd API tests
  run: dredd docs/openapi.yaml http://localhost:8080

CI Integration

# .github/workflows/docs.yml (relevant excerpt)
jobs:
  test-examples:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.12'
      - name: Install dependencies
        run: pip install -r requirements.txt
      - name: Start service
        run: docker compose up -d && sleep 5
      - name: Test examples
        run: make test-docs-examples
      - name: Link check
        run: npx --yes @umbrellio/check-links docs/

Failure Handling Policy

When an example fails in CI:

1. The PR is blocked — a failing example is a bug in the docs, not a documentation style issue.
2. The fix must come before the merge — no exceptions for "we'll fix the docs later."
3. If an example is intentionally broken (illustrating an error condition), mark it explicitly:

   # This example deliberately raises an error:
   # >>> connect_without_auth()
   # AuthenticationError: No credentials provided
   

   Wrap it in an expect-failure harness or exclude it from the test runner with a comment.

Pitfalls

• Inline copy-pasted examples that drift from the actual API — fix by using Pattern 1 (include from file).
• Testing examples only against a mock/stub service — if the mock is out of date, examples can pass and still be wrong for real users. Run against a real service in at least one CI stage.
• Letting the example test job be advisory (allowed to fail) — the moment it's advisory, nobody fixes it and it becomes noise within weeks.
• Long-running setup in examples — each example should set up and tear down its own state; shared state between examples makes failures hard to diagnose.

See Also

• ../../agents/docs-site-engineer.md — CI build/deploy, link-checking, and example testing wiring
• ../../agents/api-reference-writer.md — runnable examples in API reference