provider-routing-scaffold

name	provider-routing-scaffold
description	Scaffold a provider abstraction layer for agent cognition with mock default, per-agent routing, secret-safe logging, failure hardening, and validation tests
source	auto-skill
extracted_at	2026-05-30T05:22:10.682Z
updated_at	2026-05-30T05:33:00.000Z

Provider Routing Scaffold

When building agent-based simulations that will eventually use real LLM providers (NVIDIA NIM, OpenAI, etc.), scaffold the provider layer early with mock mode as the default. This keeps the project runnable without API keys while preparing the architecture for real integration.

Procedure

1. Create the provider interface

backend/providers/
├── __init__.py
├── base.py          # BaseProvider, MockProvider, NvidiaNimProvider, test providers
└── validation.py    # ProviderOutputValidator, ProviderOutputQuarantine

base.py contains:

• BaseProvider — abstract class with generate(prompt, agent, tick) -> str
• MockProvider — returns valid JSON {"thought": ..., "action": ...}, no external calls, default for development
• NvidiaNimProvider (or other real provider) — placeholder that reads key from env but does NOT make real calls until _real_mode = True
• ProviderCallLog — structured call logger that records provider name, agent, tick, success, latency_ms — never logs API keys or auth headers
• ProviderOutputQuarantine — quarantines invalid provider outputs to data/quarantine_provider_outputs.jsonl

Test providers (Phase 1C):

• FailingProvider — always raises RuntimeError, tests fallback behavior
• MalformedProvider — returns invalid JSON, missing fields, empty, or null fields
• BoundaryViolatingProvider — tries to force CANON_ANCHOR without source_anchor

2. Wire providers into agents with fallback

Each agent gets a provider: BaseProvider field and a set_provider() method. The agent's decide() method uses _safe_generate():

def _safe_generate(self, prompt, world):
    try:
        raw_response = self.provider.generate(prompt, self.name, self.tick)
    except Exception as e:
        # Provider failed — fallback to safe mock
        logger.warning("Provider %s failed for %s tick %s: %s", ...)
        call_log.record(provider=..., agent=..., tick=..., success=False, error=...)
        return self._mock_think_action(world) + (None, "Provider failed — fallback.")

    # Validate output
    parsed = validator.validate_and_parse(raw_response, self.provider.name, self.name, self.tick)
    if parsed["valid"]:
        return parsed["thought"], parsed["action"], parsed["label_suggestion"], parsed["simulation_note"]

    # Output invalid — fallback
    logger.warning("Provider %s returned invalid output: %s", ...)
    return self._mock_think_action(world) + (None, f"Invalid output — fallback. Reason: {parsed['repair_reason']}")

Critical: Fallback actions must use generic language that avoids canon keywords (no "creature", "boundary", "command", "tree", "garden") to prevent accidental CANON_ANCHOR classification by the consequence engine.

3. Add output validation

providers/validation.py — ProviderOutputValidator:

1. Parse JSON — quarantine if invalid
2. Check thought and action fields — quarantine if missing
3. Block CANON_ANCHOR without source_anchor — override to SIMULATION_EVENT + quarantine

4. Add config system

backend/config.py — GenesisConfig dataclass with:

• provider_mode: str — global mode ("mock", "nim-dry-run", "nim-live-disabled", "nim-live-adam-single"), default "mock"
• adam_provider: str, eve_provider: str — per-agent provider selection
• adam_nim_model: str, eve_nim_model: str — per-agent NIM model selection
• get_api_key(agent) — reads from env var, returns empty string if not set, never logs the key
• get_agent_config(agent) — returns AgentProviderConfig with provider, model, key_env
• validate() — returns warnings (e.g., "NIM mode but key not set")
• from_env() — loads from os.environ with defaults

5. Wire providers in setup function — include ALL NIM modes

Critical: The _setup_providers() function must include ALL NIM modes in its condition, not just dry-run:

def _setup_providers() -> None:
    adam_cfg = config.get_agent_config("Adam")
    if adam_cfg.provider in ("nim-dry-run", "nim-live-disabled", "nim-live-adam-single"):
        adam.set_provider(NvidiaNimProvider(
            name="adam_nim",
            api_key_env=config.nvidia_nim_key_adam_env,
            model=config.adam_nim_model,
            base_url=config.nvidia_nim_base_url,
            mode=adam_cfg.provider,
        ))
    else:
        adam.set_provider(MockProvider(name="adam_mock"))

If you add a new NIM mode and forget to add it to this condition, the agent will silently fall back to mock — the live call will never happen.

6. Environment and git protection

.env.example — variable names only, no real values:

GENESIS_PROVIDER_MODE=mock
ADAM_PROVIDER=mock
EVE_PROVIDER=mock
NVIDIA_NIM_KEY_ADAM=
NVIDIA_NIM_KEY_EVE=

.gitignore — protect:

.env
.env.local
.env.production
*.key
*.pem
*.secret
secrets/
data/private/

7. Provider call logging

Every generate() call logs:

• Provider name
• Agent name
• Tick number
• Success/failure
• Latency (ms)

Never log: API keys, auth headers, raw request/response bodies.

8. Validation tests

tests/test_provider_routing.py — Phase 1B:

1. Mock mode runs N ticks successfully
2. Adam and Eve each route through their configured provider (check call log)
3. No external API calls happen in mock mode
4. No secrets appear in logs
5. Event provenance validation still passes

tests/test_provider_failure.py — Phase 1C:

1. Failing provider does not crash tick loop
2. Malformed provider does not corrupt world state
3. Boundary-violating provider cannot create invalid CANON_ANCHOR
4. Fallback action is clearly labeled SIMULATION_EVENT
5. No secrets appear in provider logs
6. 20 normal mock ticks still pass with empty quarantine

9. API endpoint

Add /api/providers endpoint returning:

• Current config (provider_mode, per-agent providers, is_real_mode)
• Call log summary (total calls, successes, failures, by_provider)
• Recent calls (last 10)

Key Rules

• Mock is default — GENESIS_PROVIDER_MODE=mock always
• No real calls in placeholder mode — NvidiaNimProvider._real_mode = False means it returns mock responses
• Keys from env only — os.environ.get(), never hardcoded
• Keys never logged — boolean checks only (if not key: warn())
• Per-agent isolation — each agent has its own provider and its own key env var
• Fallback actions avoid canon keywords — generic text prevents misclassification
• Two-layer validation — provider output validated before simulation, events validated before log
• Quarantine invalid outputs — both provider outputs and events go to quarantine files
• Quarantine files exist — create empty files so tests can check known paths