single-live-call-hardening

name	single-live-call-hardening
description	Safely introduce exactly one live LLM API call into a running simulation with hard limits, operator checkpoint, batch blocking, and fallback validation
source	auto-skill
extracted_at	2026-05-30T06:06:53.500Z

Single Live Call Hardening

When moving from dry-run/mock to a real live LLM API call inside a simulation, introduce exactly one live call with hard limits, manual operator approval, and batch blocking. Never switch multiple agents live at once.

Provider Mode Progression

mock → nim-dry-run → nim-live-disabled → nim-live-adam-single → nim-live-eve-single → nim-live-both

Each step is validated independently before proceeding to the next.

Procedure

1. Add the single-live provider mode

PROVIDER_MODES = ("mock", "nim-dry-run", "nim-live-disabled", "nim-live-adam-single")

2. Implement live call with hard limit

class NvidiaNimProvider(BaseProvider):
    def __init__(self, ..., mode="nim-dry-run"):
        self._mode = mode
        self._live_call_count = 0
        self._max_live_calls = 1 if mode == "nim-live-adam-single" else 0

    def generate(self, prompt, agent, tick):
        api_key = os.environ.get(self._api_key_env, "")

        if self._mode == "nim-live-adam-single":
            return self._live_single_call(prompt, agent, tick, api_key)
        # ... other modes

    def _live_single_call(self, prompt, agent, tick, api_key):
        # Enforce hard limit
        if self._live_call_count >= self._max_live_calls:
            call_log.record(..., success=False, error="Live call limit reached")
            return json.dumps({"thought": "[live-limit-reached] ...", "action": "..."})

        if not api_key:
            call_log.record(..., success=False, error="No API key configured")
            return json.dumps({"thought": "[live-no-key] ...", "action": "..."})

        # Build payload
        payload = {
            "model": self._model,
            "messages": [{"role": "user", "content": prompt}],
            "temperature": 0.7,
            "max_tokens": 500,
            "response_format": {"type": "json_object"},
        }

        # Make live HTTP call (using urllib.request or httpx)
        try:
            url = f"{self._base_url}/chat/completions"
            data = json.dumps(payload).encode("utf-8")
            req = urllib.request.Request(url, data=data, method="POST")
            req.add_header("Content-Type", "application/json")
            req.add_header("Authorization", f"Bearer {api_key}")

            with urllib.request.urlopen(req, timeout=30) as response:
                raw = response.read().decode("utf-8")
                result = json.loads(raw)

            # Extract content
            content = result["choices"][0]["message"]["content"]

            # Log redacted success
            self._live_call_count += 1
            call_log.record(..., success=True, latency_ms=...)
            logger.info("NIM live call: agent=%s model=%s latency_ms=%.0f tokens=%s live_calls=%d/%d", ...)

            return content

        except Exception as e:
            call_log.record(..., success=False, error=f"Live call failed: {type(e).__name__}")
            logger.error("NIM live call failed: agent=%s tick=%s error=%s", ...)
            raise  # Re-raise so agent fallback handles it

3. Print operator checkpoint before live call

def print_operator_checkpoint(self):
    print("\n" + "=" * 60)
    print("LIVE_CALL_ARMED:")
    print(f"  agent: Adam")
    print(f"  provider: NVIDIA NIM")
    print(f"  mode: {self._mode}")
    print(f"  model: {self._model}")
    print(f"  max_calls: {self._max_live_calls}")
    print(f"  live_calls_used: {self._live_call_count}")
    print(f"  eve_provider: mock")
    print(f"  batch_ticks: disabled")
    print(f"  operator_approval_required: true")
    print("=" * 60)

4. Block batch ticks in live mode

API server:

@app.post("/api/run")
def run_ticks(count: int = 5):
    if config.provider_mode == "nim-live-adam-single":
        return {
            "status": "blocked",
            "message": "Batch ticks disabled in nim-live-adam-single mode.",
        }
    # ... normal batch logic

UI:

// In refresh(), check _live_mode flag from /api/state
if (state._live_mode) {
    document.getElementById('btn-run5').disabled = true;
    document.getElementById('btn-run20').disabled = true;
}

State endpoint:

@app.get("/api/state")
def get_world_state():
    return {
        ...,
        "_live_mode": config.provider_mode == "nim-live-adam-single",
    }

5. Live response validation

Live responses go through the same validation pipeline as mock/dry-run:

# In agent._safe_generate():
raw_response = self.provider.generate(prompt, self.name, self.tick)
parsed = validator.validate_and_parse(raw_response, ...)

if parsed["valid"]:
    return parsed["thought"], parsed["action"], ...

# Invalid live output → fallback to mock
return self._mock_think_action(world) + (None, f"Invalid live output — fallback.")

6. Redacted live-call logging

logger.info(
    "NIM live call: agent=%s model=%s latency_ms=%.0f "
    "prompt_tokens=%s completion_tokens=%s total_tokens=%s "
    "key_present=true live_calls=%d/%d",
    agent, self._model, elapsed_ms,
    usage.get("prompt_tokens"),
    usage.get("completion_tokens"),
    usage.get("total_tokens"),
    self._live_call_count, self._max_live_calls,
)

Never log: full API key, auth header value, raw response body.

7. Live-run proof endpoint

@app.get("/api/live-proof")
def get_live_proof():
    live_calls = [c for c in call_log.calls if "live" in c.get("provider", "").lower()]
    return {
        "provider_summary": {
            "mode": config.provider_mode,
            "total_live_calls": len(live_calls),
            "live_successes": len([c for c in live_calls if c["success"]]),
            "live_failures": len([c for c in live_calls if not c["success"]]),
        },
        "event_provenance": event_log.summary(),
        "quarantine_summary": {
            "events": len(event_log.quarantined),
            "provider_outputs": output_quarantine.count(),
        },
        "live_call_details": live_calls[-5:],
    }

8. Tests

def test_live_mode_builds_and_sends_request():
    provider = NvidiaNimProvider(mode="nim-live-adam-single", api_key_env="TEST_KEY")
    os.environ["TEST_KEY"] = "sk-test-key"

    with patch("urllib.request.urlopen") as mock_urlopen:
        mock_urlopen.return_value = _mock_nim_response()
        response = provider.generate("test", "Adam", 1)

    assert mock_urlopen.called, "urlopen should be called"
    assert provider.live_call_count == 1
    del os.environ["TEST_KEY"]

def test_batch_ticks_blocked_in_live_mode():
    provider = NvidiaNimProvider(mode="nim-live-adam-single", ...)
    # First call succeeds
    provider.generate("test", "Adam", 1)
    assert provider.live_call_count == 1

    # Second call blocked
    response = provider.generate("test", "Adam", 2)
    assert "limit" in json.loads(response)["thought"].lower()
    assert provider.live_call_count == 1  # Still 1

def test_eve_stays_mock_in_live_mode():
    adam.set_provider(NvidiaNimProvider(mode="nim-live-adam-single", ...))
    eve.set_provider(MockProvider(name="eve_mock"))
    # Run tick — verify Adam live, Eve mock
    assert isinstance(eve.provider, MockProvider)

Key Rules

• One agent live at a time — Adam first, then Eve, never both together initially
• Hard limit of 1 call — _max_live_calls = 1, enforced in code
• Batch ticks blocked — /api/run returns "blocked" in live-single mode
• UI buttons disabled — batch tick buttons grayed out when _live_mode is true
• Operator checkpoint printed — explicit confirmation before any live call
• Same validation pipeline — live responses go through the same validator as mock
• Fallback on failure — invalid live output triggers mock fallback, labeled SIMULATION_EVENT
• No secrets in logs — key_present=true only, never the key value
• Mock still works after live — 20-tick mock test passes after live test completes

Anti-Patterns

• ❌ Switching both agents live at once
• ❌ Allowing batch ticks in live-single mode
• ❌ Not enforcing the hard call limit in code
• ❌ Logging API keys or auth headers
• ❌ Skipping validation for live responses
• ✅ Print operator checkpoint before every live call
• ✅ Verify mock mode still works after live test
• ✅ Use /api/live-proof to audit live call history
• ✅ Proceed to Eve-only live only after Adam-only live passes