derive-tests

name	derive-tests
description	Derive obligation tests using the test harness. First checks for harness availability, then generates compact tests using domain-specific test environments. Hard gate: no harness -> no tests (stops immediately). Replaces obligation-test for harness-enabled domains.
args	<obligation-ids-or-prefix> [--count N]

Harness-Based Obligation Test Derivation

Write one behavioral test per obligation using the test harness (tests/harness/). Tests use domain-specific test environments instead of inline @patch decorators, producing 10-15 line tests instead of 40-50 line tests.

Args

/derive-tests UC-004-MAIN-01 UC-004-MAIN-02
/derive-tests UC-004 --count 10
/derive-tests UC-001-MAIN --count 15

Direct IDs: Space-separated obligation IDs. Prefix mode: Auto-selects uncovered obligations matching the prefix.

Available Harness Environments

Domain	Env Class	Tests	Production Function
Delivery poll	DeliveryPollEnv	test_delivery_poll_behavioral.py	_get_media_buy_delivery_impl
Webhook delivery	WebhookEnv	test_delivery_webhook_behavioral.py	deliver_webhook_with_retry
Circuit breaker	CircuitBreakerEnv	test_delivery_service_behavioral.py	WebhookDeliveryService, CircuitBreaker
Creative sync	CreativeSyncEnv	test_creative_sync_behavioral.py	_sync_creatives_impl
Creative list	CreativeListEnv	test_creative_list_behavioral.py	_list_creatives_impl
Creative formats	CreativeFormatsEnv	test_creative_formats_behavioral.py	_list_creative_formats_impl

Multi-transport support: Creative envs support call_via(transport) dispatch across IMPL, A2A, and REST transports. See Multi-Transport Pattern.

Protocol

Step 0: Resolve obligation IDs

If args look like a prefix (no trailing -NN sequence number):

python3 -c "
import json
al = json.loads(open('tests/unit/obligation_coverage_allowlist.json').read())
matches = sorted(oid for oid in al if oid.startswith('{prefix}'))
print(' '.join(matches[:N]))
print(f'Total matching: {len(matches)}, selected: {min(N, len(matches))}')
"

Step 1: Harness gate (HARD STOP)

Check if a harness exists for the target domain:

ls tests/harness/*.py

Map obligation prefix to required harness. For the complete, auto-generated list of all env classes with their methods, read .agent-index/harness/envs.pyi.

If harness missing: STOP IMMEDIATELY. Print:

No harness found for {domain}. Create tests/harness/{domain}.py first.
See tests/harness/_base.py for the IntegrationEnv pattern.

Do NOT fall back to inline mocking. Generating tests without a harness produces architecturally poor code that wastes tokens.

If harness exists: CONTINUE.

Step 2: Read harness API + gold standard

1. Read the harness class for available methods:

   # Example for delivery poll:
   head -60 tests/harness/delivery_poll.py
   

   Extract: class docstring, fluent API methods, call_impl signature.

2. Read the meta-test file for usage examples:

   # Example:
   cat tests/harness/test_harness_delivery_poll.py
   

   These are your gold standard — tests use the exact same pattern.

3. Read 1-2 converted tests in the target file for style reference:

   # Look for tests already using the harness:
   grep -A 15 "DeliveryPollEnv\|WebhookEnv\|CircuitBreakerEnv" tests/unit/test_delivery_*_behavioral.py
   

Step 3: Research obligation

For each obligation:

1. Read the scenario from docs/test-obligations/:

   grep -A 10 "{OID}" docs/test-obligations/*.md
   

   Extract: Given/When/Then, business rule, priority, layer.

2. Translate Given/When/Then directly into the test:

   • Given → test setup (fixtures, env configuration)
   • When → action (call production function)
   • Then → assertions (expected output/state)

   The BDD spec is the sole source of expected behavior. Do NOT derive assertions from what the production code currently does. If the spec says "Then a SyncCreativesSubmitted is returned," assert isinstance(result, SyncCreativesSubmitted) — even if the code currently returns something else or doesn't implement the behavior at all.

3. Check if production code implements it: Locate the _impl function.

   • Implemented → test should PASS. If it doesn't, the code has a bug.
   • Not implemented → test MUST still assert spec behavior, marked with @pytest.mark.xfail(strict=True, reason="<what's missing>"). Never write a test that asserts current (wrong) behavior just because the spec behavior doesn't exist yet. That legitimizes the gap.

Step 4: Write test

Write ONE test following all 7 hard rules:

#	Rule	Check
1	Import from src.	from src. in test file
2	Call production function	Test calls env.call_impl() or production function
3	Assert production output	Assertion checks a value from the production call
4	Covers: {OID} tag	Docstring contains exactly Covers: {OID}
5	Use harness env	Test uses harness context manager, not inline @patch
6	Not mock-echo only	Does more than verify mock.called
7	No inline @patch	MUST NOT define @patch decorators or inline UoW setup

Test template (delivery poll — integration, preferred):

import pytest
from tests.factories import TenantFactory, PrincipalFactory, MediaBuyFactory

@pytest.mark.requires_db
class TestObligationName:
    """Short description of what's being tested.

    Covers: {OID}
    """

    def test_specific_behavior(self, integration_db):
        """What specific behavior is verified.

        Covers: {OID}
        """
        from tests.harness import DeliveryPollEnv

        with DeliveryPollEnv() as env:
            tenant = TenantFactory(tenant_id="t1")
            principal = PrincipalFactory(tenant=tenant, principal_id="p1")
            buy = MediaBuyFactory(tenant=tenant, principal=principal)
            env.set_adapter_response(buy.media_buy_id, impressions=5000)
            response = env.call_impl(media_buy_ids=[buy.media_buy_id])
            assert response.some_field == expected_value

Test template (delivery poll — unit, backward compat):

class TestObligationName:
    """Short description.

    Covers: {OID}
    """

    def test_specific_behavior(self):
        """What is verified.

        Covers: {OID}
        """
        from tests.harness.delivery_poll_unit import DeliveryPollEnv

        with DeliveryPollEnv() as env:
            env.add_buy(media_buy_id="mb_001", ...)
            env.set_adapter_response("mb_001", impressions=5000)
            response = env.call_impl(media_buy_ids=["mb_001"])
            assert response.some_field == expected_value

Test template (webhook):

class TestObligationName:
    """Short description.

    Covers: {OID}
    """

    def test_specific_behavior(self):
        """What is verified.

        Covers: {OID}
        """
        from tests.harness import WebhookEnv

        with WebhookEnv() as env:
            env.set_http_status(503, "Unavailable")
            success, result = env.call_deliver(max_retries=3)
            assert success is False
            assert result["attempts"] == 3

Test template (circuit breaker):

class TestObligationName:
    """Short description.

    Covers: {OID}
    """

    def test_specific_behavior(self):
        """What is verified.

        Covers: {OID}
        """
        from tests.harness import CircuitBreakerEnv

        with CircuitBreakerEnv() as env:
            breaker = env.get_breaker(failure_threshold=3)
            for _ in range(3):
                breaker.record_failure()
            assert breaker.state == CircuitState.OPEN

Step 5: Run and fix

uv run pytest <test_file>::<TestClass>::<test_method> -x -v

• PASS or XFAIL = proceed
• ERROR = fix (import/name errors, wrong harness method)
• If the harness doesn't expose what you need, use env.mock["name"] to access the underlying mock directly

Step 6: Verify

Six mechanical checks:

1. grep "from src\." <file> > 0
2. grep "Covers: {OID}" tests/ count == 1
3. Test runs without ERROR
4. make quality passes
5. No duplicate Covers tags
6. If test PASSES: remove OID from allowlist, add file to _UNIT_ENTITY_FILES if needed, run obligation guard

Step 7: Commit

git add <test_file> tests/unit/obligation_coverage_allowlist.json
git add tests/unit/test_architecture_obligation_coverage.py  # if changed
git commit -m "test: add obligation test for {OID}"

Multi-Transport Pattern

When an obligation has transport variants in upstream BDD (e.g., both T-UC-006-main-rest and T-UC-006-main-mcp), generate ONE parametrized test instead of two separate tests.

Decision rule:

1. Check if upstream BDD duplicates the scenario across REST/MCP transports
2. If yes → parametrize with ALL_TRANSPORTS
3. If no → use single transport (Transport.IMPL)

Template (multi-transport — creative domains):

import pytest
from tests.factories import TenantFactory, PrincipalFactory
from tests.harness import CreativeSyncEnv, Transport, assert_envelope

ALL_TRANSPORTS = [Transport.IMPL, Transport.A2A, Transport.REST]

@pytest.mark.requires_db
class TestObligationName:
    """Short description.

    Covers: T-UC-006-xxx-rest, T-UC-006-xxx-mcp
    """

    @pytest.mark.parametrize("transport", ALL_TRANSPORTS, ids=lambda t: t.value)
    def test_specific_behavior(self, integration_db, transport):
        """What is verified.

        Covers: T-UC-006-xxx-rest, T-UC-006-xxx-mcp
        """
        with CreativeSyncEnv() as env:
            # SHARED FIXTURE
            TenantFactory(tenant_id="test_tenant")
            PrincipalFactory(tenant_id="test_tenant", principal_id="test_principal")

            # TRANSPORT DISPATCH
            result = env.call_via(transport, creatives=[...])

        # ENVELOPE (transport-specific)
        if transport == Transport.REST:
            assert_envelope(result, Transport.REST)

        # PAYLOAD (shared — identical for all transports)
        assert result.is_success
        assert result.payload.creatives[0].action == expected_action

Transport-specific behaviors (e.g., REST returns 401, MCP raises ToolError) get separate non-parametrized tests — don't mix them into the shared test.

Creative Harness API

from tests.harness import CreativeSyncEnv, CreativeListEnv, CreativeFormatsEnv, Transport

# Sync creatives (multi-transport)
with CreativeSyncEnv() as env:
    env.set_registry_formats([...])
    result = env.call_via(Transport.REST, creatives=[...], dry_run=True)
    # result.is_success, result.payload, result.envelope

# List creatives (multi-transport)
with CreativeListEnv() as env:
    result = env.call_via(Transport.A2A, media_buy_id="mb_001")

# List formats (multi-transport)
with CreativeFormatsEnv() as env:
    env.set_registry_formats([...])
    result = env.call_via(Transport.IMPL)

All three envs support: call_via(transport), call_impl(), call_a2a(), build_rest_body(), parse_rest_response().

Test File Selection

Domain	File
UC-004 delivery poll	test_delivery_poll_behavioral.py
UC-004 webhook	test_delivery_webhook_behavioral.py
UC-004 service/CB	test_delivery_service_behavioral.py
UC-002	test_create_media_buy_behavioral.py
UC-003	test_update_media_buy_behavioral.py
UC-006 sync (multi-transport)	test_creative_sync_transport.py
UC-006 sync (impl only)	test_creative_sync_behavioral.py
UC-006 list (impl only)	test_creative_list_behavioral.py
UC-006 formats (impl only)	test_creative_formats_behavioral.py

New files must be added to _UNIT_ENTITY_FILES in tests/unit/test_architecture_obligation_coverage.py.

Harness API Quick Reference

DeliveryPollEnv

from tests.harness import DeliveryPollEnv

with DeliveryPollEnv(principal_id="p1", tenant_id="t1") as env:
    # Add mock media buys to the repository