agent-ops-debugging

name	agent-ops-debugging
description	Diagnose agent operational failures through default-policy checks, multi-gap analysis, explicit service ownership, request-context lifetime, and degradation-versus-correctness checks.

Agent Ops Debugging — Autonomous Agent Operational Debugging Patterns

Source: Distilled from GEODE operational debugging sessions (2026-03-14) Philosophy: "Not crashing" and "working correctly" are different problems. Historical incident: ADR-008. The referenced Blog 25 (25-operational-debugging-four-layer-fix.md) is absent from this checkout; it is retained here only as historical provenance. Current contract: construction and lifecycle.

5 Pattern Overview

#	Pattern	One-line Principle	Applicable Layer
D1	Safe Default Anti-pattern	Safe defaults guarantee stability but not correctness	All layers
D2	Multi-gap Root Cause	Individual gaps are harmless but N gaps combined manifest — individual tests cannot catch them	Pipeline
D3	Ownership and Context Lifetime	Compose services explicitly; bind and reset request-local context	Wiring + Runtime
D4	Execution Boundary	Choose explicit snapshots or context propagation according to ownership	Runtime + Concurrency
D5	Degradation ≠ Correctness	Graceful skip/fallback and functional correctness must be verified separately	Verification

---

D1. Safe Default Anti-pattern

Principle

When defaults are set to safe values like True, None, or "", the system does not crash but returns results different from the intended behavior. This type of bug does not appear in error logs, so it is discovered late.

Diagnostic Criteria

Q: If this code path operates with the default value, is the result "normal" or "degraded"?

Default Type	Normal Case	Degraded Case
dry_run=True	Returns fixture when no API key	Returns only fixture even with API key present
return None	Optional feature not in use	Required feature silently disabled
log.debug(skip)	Optional external integration skipped	Required integration silent fail

Application Pattern

# BAD — always safe but always degraded
dry_run = args.get("dry_run", True)

# GOOD — default determined by system state
dry_run = args.get("dry_run", force_dry_run)  # force_dry_run is readiness-based

Use the owner's declared default and distinguish normal, degraded, and failed outcomes. The example above describes the 2026-03-14 incident; credential availability alone does not authorize live calls or changing a dry-run request.

---

D2. Multi-gap Root Cause Analysis

Principle

The most difficult operational bugs to find are those that manifest only when N independent gaps exist simultaneously. Each gap is individually harmless or has separate safeguards, so they are not discovered by unit tests.

Analysis Framework

1. Symptom definition: "On which path, under which conditions, does the result differ from expectations"
2. Path comparison: "Identify divergence points compared to the normally working path"
3. Gap enumeration: "Verify whether each divergence point has an independent gap"
4. Overlap determination: "Must all gaps exist simultaneously for the bug to manifest?"

Case: Sub-agent dry-run (3-gap)

Gap 1: Handler default = True (hardcoded)
  → Alone: Solvable if LLM passes dry_run=False
Gap 2: dry_run not defined in tool schema
  → Alone: LLM cannot know the parameter
Gap 3: ContextVar thread isolation
  → Alone: Handler cannot query readiness

All 3 present → sub-agent path always runs in dry-run

Fix the smallest root cause, then verify whether the remaining gaps are actually harmless. Do not close them from the existence of one fix alone.

---

D3. Service Ownership and Request Context

Principle

Current GEODE composes services in core.wiring through constructors or existing typed contexts. Do not restore the historical ContextVar service locator. ContextVars may hold request identity, diagnostics, or request-local state; their binding and reset must match the logical request lifetime.

Entry Point Checklist

Typical entry points for an agent system:
[ ] CLI single command
[ ] REPL interactive loop (e.g., `geode` → interactive)
[ ] Hosted runtime turn
[ ] Sub-agent task, worker thread, or subprocess
[ ] Server or scheduled-job entry point
[ ] Test fixture (e.g., pytest conftest.py)

For the affected entry points, trace the service constructor to its consumer and the request-context binder to its reset, including errors and cancellation. Reuse the existing composition owner; entry-point count alone does not justify another bootstrap layer or DI container.

---

D4. Crossing Task, Thread, and Process Boundaries

Principle

Check how the actual execution primitive propagates context. A copied context does not synchronize mutable services, and a captured value can become stale. Pass services explicitly and choose the lifetime of value snapshots deliberately.

Historical pattern — ADR-008, 2026-03-14

def make_handler(*, force_dry_run: bool = True):
    """force_dry_run is captured in the closure — same regardless of which thread executes."""
    def handler(task_type: str, args: dict) -> dict:
        dry_run = args.get("dry_run", force_dry_run)
        ...
    return handler

# Historical call site: fix readiness state at handler creation time
readiness = _get_readiness()
handler = make_handler(force_dry_run=readiness.force_dry_run)

Current decision criteria

Value	Boundary treatment	Verify
Service dependency	Constructor or existing typed context	Correct owner and teardown
Immutable per-turn policy	Explicit snapshot at the owning turn boundary	A later config change affects the intended next turn
Request identity or diagnostics	Scoped binding or explicit context propagation	Correlation preserved; reset on success/error/cancel
Cross-process input	Existing serialized request contract	No assumption that in-process ContextVars cross the boundary

The historical closure fixed one readiness lookup. It is not a recommendation to capture mutable credentials or long-lived policy at handler construction.

---

D5. Degradation ≠ Correctness

Principle

Graceful degradation is a system stability pattern. It must be verified separately from functional correctness.

Verification Matrix

For all external dependencies:

| Dependency | Expected behavior when present | Expected behavior when absent | Actual behavior |
|------------|-------------------------------|-------------------------------|-----------------|
| Provider   | authorized call uses selected source | explicit error or explicitly requested fixture mode | ??? |
| MCP        | tool list loaded              | skip + warning                | ???             |
| Redis      | L1 cache used                 | direct L2 query               | ???             |

Test both columns with offline fakes first. Live verification requires the task's authorization; merely finding credentials does not authorize it.

Distinction Criteria

Stability test: "Does the system not crash when dependency X is absent?"
Correctness test: "Does the system actually use X when dependency X is present?"

Verify the declared outcome in both cases. A required dependency may correctly fail closed instead of returning a successful-looking fallback.

---

Debugging Workflow

When encountering a symptom of "it works but not correctly" during operations:

1. Symptom → Identify the affected layer (infrastructure/UI/DI/pipeline)
2. Compare with the normal path to find divergence points
3. Check defaults at divergence points — D1 Safe Default applicable?
4. Determine single cause vs Multi-gap — D2 applicable?
5. ContextVar access failure? — D3/D4 applicable?
6. After fix, verify both "when present" + "when absent" — D5