testing

name	testing
description	How to write and run fast, deterministic, behavior-focused pytest tests that never touch real external systems. Use whenever adding or changing behavior, or fixing a bug.
generated	true
source	.ai/skills/testing.md

Skill: Testing

Purpose

Define how to write and run tests for this Python project so that every meaningful behavior is covered by fast, deterministic, behavior-focused tests that never touch real external systems.

When to Use This Skill

• Adding or changing any behavior (tests accompany behavior — almost always).
• Fixing a bug (add a regression test first).
• Working on the test suite or quality gate (task 0012).
• Reviewing whether a change is adequately tested.

Files to Read First

• AGENTS.md
• docs/DEVELOPMENT_WORKFLOW.md
• pyproject.toml ([tool.pytest.ini_options])
• The skill for the area under test (runtime, prompts, plugins, tools, etc.).

Core Principles

• Use pytest. Tests live under tests/, mirroring src/agentplatform/.
• Test behavior through public interfaces, not private implementation details (test internals only when there is no public seam).
• No real external systems in unit tests: never call real LLMs, real MCP servers, real databases, or third-party APIs. Mock/fake them.
• Integration tests only when explicitly requested, and still without real secrets or live third-party calls — use fakes/local stand-ins.
• Deterministic: control time, randomness, and ordering.
• Regression tests for bugs: every fixed bug gets a test that fails before the fix.

Process

1. Pick the category (see below) for what you are testing.
2. Arrange with fixtures for repeated setup (specs, compiled graphs, fake resolver/access plugins, fake tools). Use @pytest.mark.parametrize for input variations.
3. Mock external boundaries (LLM/MCP/DB/HTTP) at the adapter seam, not deep inside.
4. Write behavior assertions: given input, assert observable output/effects.
5. Cover negatives and security: invalid YAML/validation errors, missing required prompt variables, denied protected-node access, plugin errors.
6. Run make test (and make check for the full gate). Fix failures.

Testing categories (use the right one)

• Unit tests — a single function/class via its public interface.
• Integration tests — multiple layers together (e.g. validate → compile → run) with fakes; only when requested.
• Contract tests — verify plugin contracts and the YAML schema match docs/; catch drift early.
• Golden example tests — a known YAML config produces a known compiled graph / rendered prompt; update goldens deliberately.
• Negative tests — invalid specs, missing variables, denied access must fail clearly with the right error.
• Security tests — protected access fail-closed, secrets redacted, no request-state leakage.

Example pytest structure (illustrative, not product tests)

import pytest

@pytest.fixture
def valid_spec_dict() -> dict:
    return {"version": "1.0", "app": {"name": "demo"}, "...": "..."}

@pytest.mark.parametrize("missing_key", ["version", "app", "runtime"])
def test_validation_reports_missing_top_level_key(valid_spec_dict, missing_key):
    data = dict(valid_spec_dict)
    del data[missing_key]
    result = validate_spec(data)            # public interface
    assert not result.ok
    assert any(missing_key in e.message for e in result.errors)

Do not implement real product tests in this task. Only trivial repository smoke tests (e.g. import + version) belong here until task 0001+ create code.

Checklist Before Finishing

• New/changed behavior has tests; bugs have regression tests.
• Tests target public interfaces and assert behavior, not internals.
• External systems are mocked/faked; no real LLM/MCP/DB/API calls.
• Negative and security cases covered where relevant.
• Tests are deterministic (time/randomness/order controlled).
• Fixtures/parametrization used to keep tests readable and DRY.
• make test (and make check) pass, or you state why they can't run.

Common Mistakes to Avoid

• Calling real LLMs/MCP/DBs/APIs in unit tests.
• Asserting on private attributes/log strings instead of behavior.
• Flaky tests depending on real time, randomness, or ordering.
• Over-mocking until the test asserts nothing meaningful.
• Skipping negative/security tests because the happy path passes.
• Committing secrets or production endpoints in fixtures.

Expected Final Report

State: which test files/categories were added or changed; what behaviors and edge/negative/security cases are now covered; mocks/fakes used for external boundaries; the result of make test / make check; and any coverage gaps intentionally left for a later task.