daaf-deploy-smoke-testing

name	daaf-deploy-smoke-testing
description	In-situ smoke testing that verifies a LIVE DAAF deployment functions end-to-end in whatever provider route it is configured for. Auto-detects the active install route (anthropic-subscription, openrouter, chatgpt-subscription shim, openai-api shim) and runs tiered probes: Tier 0 free preflight (route/env coherence, hooks, statuslines, shim health); Tier 1 one live round-trip (~cents); Tier 2 a six-probe functional battery (dispatch, coding, web, skill loading, isolation-strip, nested-dispatch deny); Tier D a zero-cost deterministic battery (bats, Pester, lint, safety-hook tests). Use after install, after a rebuild or update, after any provider/config change (new API key, model remap, shim change), for pre-release validation, or when asked "is my install working" or to "smoke test the deployment/config." Requires the DAAF_DEV=1 dev image; a contributor tool, not end-user-facing. NOT DAAFBench (model behavioral-adherence evaluation under benchmarks/) and NOT scripts/smoke_tests/ (R/Python library-package smoke tests).
metadata	{"audience":"any-agent","domain":"research-orchestration"}

DAAF Deployment Smoke Testing

Route-aware, in-situ verification that a live DAAF installation actually works, configured exactly as the user set it up. The suite (scripts/deploy_smoke/) auto-detects which of DAAF's four provider routes is active from the live environment, then runs tiered probes — from a free no-LLM preflight up through live functional round-trips — and writes an evidence-quoted report with an overall PASS/FAIL. It answers one question: does this deployment function end-to-end in its configured route? It is deliberately scoped to deployment/configuration health, not model quality (that is DAAFBench) and not package availability (that is scripts/smoke_tests/). This is a development/contributor tool: it assumes the DAAF_DEV=1 image and is not surfaced in the end-user install guide. Triggers include post-install verification, post-rebuild/update checks, any provider or configuration change, pre-release validation across multiple configurations, and direct requests like "is my install working" or "smoke test the config."

Critical Disambiguation — Three Smoke Tools, Three Questions

All three exist in the repo; loading the wrong one wastes effort. Route by the question being asked:

What is being verified?
├─ "Does my DEPLOYMENT work in its provider route?"          → THIS skill
│    (live install health: routing, hooks, dispatch, shim, statuslines)
│    scripts/deploy_smoke/run_deploy_smoke.py
│
├─ "Does a MODEL follow DAAF's rules / adhere behaviorally?" → DAAFBench
│    (scored behavioral-adherence evaluation across a case battery)
│    benchmarks/  — heavyweight, cost-driven, NOT an install check
│
└─ "Does an R/Python LIBRARY load and run?"                  → library smokes
     (per-skill package import/execute checks)
     scripts/smoke_tests/run_all_smoke_tests.sh

The deployment suite reuses Tier D to run the library smokes and reuses DAAFBench's output-parsing helpers, but its purpose is distinct: it is the only tool that verifies the wiring between Claude Code, the configured provider, the hooks, and the subagent machinery is intact for this install.

What It Verifies (by tier)

Tier	Cost	What it checks	LLM?
0	free	Route detection + --route assertion, DAAF_DEV=1, env coherence for the detected route, model-family + ceiling-hook posture, context-window declaration (lane-aware: DAAF_PROVIDER_SHIM+SHIM_BACKEND_MODE select the ~370k chatgpt-subscription ceiling), CLI liveness, all hooks registered, both statuslines render, shim /health (shim routes), chatgpt auth.json readable, workspace invariants, R UTF-8 locale (image LANG/LC_ALL=C.UTF-8)	no
1	~cents	One live claude -p round-trip + the plumbing around it: response returned, transcript located, audit-log hook fired, context-reporter injection, /tmp coordination caches, statusline against the real session, token/cost parsing	1 call
2	~$0.60–3.00 / profile	Six-probe functional battery: subagent dispatch + search, coding agent writes + runs via run_with_capture, web access (WebSearch), skill loading, isolation-strip hook (SKIP-tolerant), nested-dispatch deny hook	6 calls
D	free	Deterministic battery: a harness self-test first (TD.0 — the suite's own provider-free unittest module), then bats tests/bash, Pester, check-daaf-conventions.sh lint, safety-hook tests, single-command hook tests, and (opt-in) the R/Python library smoke suite. Every Tier D subprocess runs under a sanitized env — the two live-config contaminants CLAUDE_CODE_MAX_CONTEXT_TOKENS and DAAF_BRANCH are stripped (defense-in-depth atop each battery's own fixture isolation); PATH/HOME/credentials/toolchain are preserved	no

Tier 2 cost scales with the configured model. Tiers 0 and D run once; Tiers 1 and 2 run once per profile. Each Tier 2 run operates inside its own per-run, UUID-owned sandbox (scripts/deploy_smoke/_sandbox/run_<uuid>/) that it self-cleans on exit, so no fixture from a prior run can satisfy a later one; correspondingly, the T2.2 coding probe PASSes only on freshly captured execution evidence, not on source text or a stale log (see references/interpreting-results.md).

When to Run

• After a fresh install or a rebuild/update — confirm the deployment still routes, dispatches, and enforces hooks.
• After any provider or configuration change — a new API key, a model remap (ANTHROPIC_DEFAULT_*), a shim backend switch, a base-URL change.
• Before a release — validate across each configuration you support (Anthropic, OpenRouter with multiple model families, the shim routes).
• When something feels off — statusline shows the wrong window, subagents behave oddly, a hook seems not to fire.

Invocation

Run from anywhere (the CLI resolves the repo root itself). All commands are single Bash calls.

# Free preflight only — no API cost, good first check after any change:
python3 /daaf/scripts/deploy_smoke/run_deploy_smoke.py --tiers 0 --yes

# Default run (Tiers 0,1,2) on the auto-detected route, with the cost confirmation prompt:
python3 /daaf/scripts/deploy_smoke/run_deploy_smoke.py

# Assert the expected route — detection mismatch is a FAIL (catches a mis-set env):
python3 /daaf/scripts/deploy_smoke/run_deploy_smoke.py --route anthropic-subscription --tiers 0

# Everything, including the deterministic battery and the (slow) library smokes:
python3 /daaf/scripts/deploy_smoke/run_deploy_smoke.py --tiers 0,1,2,D --include-r-smoke --yes

Flags (verified against --help):

Flag	Effect
--route ROUTE	Assert the expected route; a detection mismatch is a FAIL instead of a silent override. Values: anthropic-subscription, openrouter, chatgpt-subscription, openai-api.
--profiles A,B,C	Comma-separated profile names from profiles.yaml (OpenRouter route — see below).
--tiers 0,1,2,D	Which tiers to run. Default 0,1,2. Add D for the deterministic battery.
--include-r-smoke	Include the slow R/Python library smoke suite inside Tier D.
--timeout N	Per-probe timeout seconds (0 = per-tier defaults: 180/300/600).
--yes	Skip the pre-launch cost/summary confirmation (needed for non-interactive runs).
--report-dir DIR	Override the report output directory.

The suite omits --model on its claude -p probes by default, so it tests the model as configured (from settings.json / ANTHROPIC_MODEL / the profile overlay) — exactly what a real session would resolve.

OpenRouter Multi-Profile Runs

The OpenRouter route can talk to multiple model families. --profiles runs Tiers 1–2 once per named profile (Tiers 0 and D run once), applying each profile's env overlay to the probe subprocess only:

python3 /daaf/scripts/deploy_smoke/run_deploy_smoke.py --route openrouter \
    --profiles openrouter-claude,openrouter-gpt,openrouter-glm --yes

Profiles are defined in scripts/deploy_smoke/profiles.yaml (the three above are working examples — adapt the model slugs to models your OpenRouter account can reach). For non-Claude models a profile sets ANTHROPIC_DEFAULT_OPUS_MODEL / ANTHROPIC_DEFAULT_SONNET_MODEL (keeps subagents model-pure and makes enforce-model-ceiling.sh stand down) and CLAUDE_CODE_MAX_CONTEXT_TOKENS (declares the real window so context reporting is accurate).

Shim routes are different. On openai-api / chatgpt-subscription the shim daemon state (SHIM_SANITIZE_TOOLS, backend_mode) is fixed at daemon startup and cannot be changed per run — env overlays reach the CLI session, not the running shim. --profiles on a shim route emits a WARNING; verify daemon state read-only via Tier 0's shim /health probe instead.

Reading the Output

Each run writes scripts/deploy_smoke/reports/{YYYYMMDD_HHMMSS}_{route}/:

• report.md — human audit: per-probe verdict with quoted evidence, grouped by tier, plus a redacted env fingerprint.
• report.json — machine-readable: git SHA, route, model family, redacted env fingerprint, per-probe results.
• evidence/ — discrete audit-snapshot files: shim_health.json (shim routes only) and env_fingerprint.json. The /tmp coordination-cache reads a Tier 1 probe performs are embedded inline in that probe's report.md/report.json evidence — they are not written as separate files under evidence/.
• evidence/tier_d/ — Tier D failure artifacts (present when Tier D runs): the COMPLETE scrubbed output of any FAILed or timed-out battery is persisted at evidence/tier_d/{probe_id}.log, and Pester's NUnit testResults.xml is written here (report-local) rather than at the repository root. On a PASS the report keeps only a concise final excerpt; the full log is persisted only on FAIL/timeout.

Verdicts are PASS | FAIL | WARN | SKIP | INFO. Only FAIL flips the overall result and the exit code (nonzero), matching the run_all_smoke_tests.sh contract. A route-appropriate SKIP (e.g. shim /health on a non-shim route) or a tolerant WARN never breaks the run.

Interpreting individual probes — verdicts are not always self-explanatory. Some WARN/INFO signals are expected mechanics of a headless probe, not install defects (notably the Tier 1 context-reporter and /tmp-cache probes on short runs), and each route has its own known failure modes with documented fixes. For probe-by-probe interpretation, route-conditional expectations, and failure-mode → fix routing, read references/interpreting-results.md.

Route Auto-Detection (summary)

Detection reads the live environment (no LLM, no subprocess):

Live env signal	Detected route
DAAF_PROVIDER_SHIM=openai + SHIM_BACKEND_MODE=chatgpt	chatgpt-subscription (shim)
DAAF_PROVIDER_SHIM=openai (otherwise)	openai-api (shim)
ANTHROPIC_BASE_URL contains openrouter.ai	openrouter
none of the above	anthropic-subscription

The shim gate is checked before the OpenRouter base-URL test because shim routes point ANTHROPIC_BASE_URL at the localhost shim, not at openrouter.ai. --route turns your expectation into an assertion so a silently mis-set env surfaces as a FAIL.

Boundaries

• Requires DAAF_DEV=1. The deployment smoke suite is intentionally restricted to development images as a contributor tool; the Tier 0 policy gate applies even to --tiers 0. Tier D additionally relies on development-only deterministic/test tooling, although an unavailable Tier-D tool can produce a probe-level SKIP. Codex itself ships in every image. Shim routing and codex login are separate explicit opt-ins and do not require a development image. The suite remains intentionally absent from the end-user install guide.
• Secrets never leak. The env fingerprint redacts any var whose name contains KEY/TOKEN/SECRET/AUTH, but preserves the load-bearing empty-vs-unset distinction (e.g. OpenRouter needs ANTHROPIC_API_KEY present-and-empty).
• Reads /tmp caches, never writes them. Reading DAAF's coordination caches is the sanctioned pattern; the suite writes only inside the project.
• Do not confuse tiers with cost tiers. Tier 0 and Tier D are free; Tiers 1 and 2 make live API calls billed to your configured provider. The pre-launch summary quotes the call count before any live tier runs (skip it with --yes).

Reference Files

File	When to read
references/interpreting-results.md	Reading a report: probe-by-probe meaning, route-conditional expectations, headless-mode nuances, and known failure-mode → documented-fix routing per route