threadlight-safe-check

## Presenter-ready consumer gate An explicit `delivery_profile: presenter-ready` adds the [presenter-ready contract](../../docs/presenter-ready.md), consumed from a complete catalog with `skills/_shared`. Design, Deploy and this checker use the same [`presenter-deployment-pin.json`](../_shared/presenter-deployment-pin.json). Validate the selected consumer's real manifest location, protocol, environment list/map and runtime root; reject active competing manifests. This profile's native layout supersedes the legacy `agent.yaml`/mandatory-Bicep assumptions below, not existing resource, reachability or selected-governance gates. Pre-deploy validates the packaged service inventory. Post-deploy retains its live checks and separately consumes process journey evidence; a green completeness manifest does not imply script-verified or human-accepted. Use `python3 -m skills._shared.presenter --root <pilot>` for that read-only assessment. # Threadlight Safe Check — three lifecycle gates, one CLI The single mandatory completeness gate for any threadlight pilot. Replaces ad-hoc Phase 3 / Phase 3.5 checks scattered through `threadlight-deploy` with one consolidated CLI you invoke at three lifecycle points: ``` post-design → SPEC <-> manifest.json deployment_manifest contract pre-deploy → manifest <-> azure.yaml services <-> infra/main.bicep <-> src/<dir>/ post-deploy → manifest <-> az resource list <-> channel reachability ``` > **Why this skill exists.** A recent investigation-style PoC > shipped with **`aca-bot: yes`, `aca-job: yes`, `workspace-ui: yes`** > in SPEC § 11c — and zero bot resources, zero jobs, zero workspace ACAs > in the deployed resource group. `azd up` returned 0. Eval scores > looked plausible. The gap was caught by the user opening the Azure > Portal and noticing missing tiles. **A consolidated, mandatory gate > would have caught this in 30 seconds.** That gate is this skill. > > **And then it shipped again, differently.** A later pilot > using the `fetch-container-image` pattern had every resource type > present in the resource group — > `safe-check` returned `gaps: []` ✅ — but the MCP container was > running `mcr.microsoft.com/azuredocs/containerapps-helloworld:latest` > (Bicep had hard-coded the placeholder; `azd deploy mcp` was never > run after the provision), and the deadline-watcher cron had > 13 consecutive `Failed` executions. Structural-only checks aren't > enough. The post-deploy phase now also runs **behavioural** checks: > deployed images must NOT match the azuredocs placeholder regex, > and no scheduled job may have its last 5 executions all `Failed`. ## What this skill does NOT replace - **Invocation testing** of the agent → use `foundry-evals` - **Authoring** the manifest → use `threadlight-design` (its `deployment_manifest{}` JSON block in `specs/manifest.json` is the contract this skill consumes) - **Running `azd up`** → use `threadlight-deploy`. This skill is invoked **before** and **after** `azd up`, never instead of. ## When to invoke | Lifecycle point | Phase | What's checked | Gate result | |---|---|---|---| | After SPEC + AGENTS.md drafted | `--phase design` | `specs/manifest.json` contains `deployment_manifest{}`; SPEC § 11c rows match `module_selectors` keys | Drift / fail | | Before `azd up` | `--phase pre-deploy` | Every `yes` selector → wired in `azure.yaml` + `infra/main.bicep` + has `src/<dir>/Dockerfile`; no orphan Bicep / src folders | Fail-fast | | After `azd up` returns 0 | `--phase post-deploy` | Every `expected_resource_types` entry in `az resource list`; required ACA roles by name pattern; **every deployed image is the real image (NOT the azuredocs placeholder)**; **no scheduled job has its last 5 executions all `Failed`**; all `channels` reach HTTP/JWT-OK; `scheduled_jobs` cron correct | **The non-negotiable gate.** Empty `gaps[]` = PoC complete | Each phase emits a JSON manifest under `tests/` so the gate is auditable and re-readable later (CI, demo prep, postmortem): - `tests/safe-check-design-manifest.json` - `tests/safe-check-predeploy-manifest.json` - `tests/postdeploy-manifest.json` *(name preserved for backwards-compat with the prior `threadlight-deploy` Phase 3.5 manifest)* All three manifests have a top-level `"gaps": []`. **Empty array = pass.** ## Selected runtime governance The host protocol and enforcement producer are separate selections. MAF Responses can use the gateway producer for an explicitly generated gateway-only contract; it must not be treated as a local Agent Hooks producer. Static checks retain the actual MAF host/helper and staged signed gateway association. No native-probe assets or native database permissions are inferred from the framework name. This changes routing, not the proof boundary: `governance_probe_noop` never proves a business write, OBO or long-lived approval. When the SPEC selects governance, resource presence alone is insufficient. Explicit `off` and unselected legacy projects add no governance network calls. Invalid selected configuration is not off: it fails closed. Legacy v2 evidence never passes new runtime readiness. This is binding evidence, **not whole-agent** certification. - **Pre-deploy:** verify the exact selected adapter, packaged code/configuration, bundle integrity, required controls and service bindings. Before an image exists, record `pre-image` and explicit unverified image/deployment facts; never call a static declaration enforced. Intentionally unbound read tools are not missing adapters and must not be auto-selected or removed. - **Post-deploy:** use the installed collector and the operator's protected `.threadlight/governance-probe.json`. A Key Vault verified registry must explicitly declare **`probe_safe: true`** for **`governance_probe_noop`**. Without it, do not invoke the model or improvise a business operation, even with `--force`. Missing setup is an unverified governance gap. - Register fresh, independent allow/deny UUIDs at both producer and fixture before invoking the actual hosted agent. Require terminal producer states, an authenticated Task 8 receipt, one allow effect and zero deny effects. Ignored prompts, replay, unavailable dependencies, wrong scope or changed deployment remain unverified. Model text is never an oracle. - Dedicated noop proof **cannot certify business bindings** or other lifecycle points. Those remain unverified without their own explicit safe live evidence. Do not promote local tests, CTK results or packaging into deployed proof. - Preserve every existing non-governance gap. `governance_health`, `governance_probes` and `governance_gaps` supplement—not replace—`gaps`. Saved collector JSON is payload-free evidence, **not remote attestation**. See [collector installation, inputs and limits](references/governance-probe.md) and the [actual producer contract](references/probe-fixture/README.md). The collector independently observes ARM/Foundry tenant/subscription/RG, version/image/principal and closed configuration before and after invocation. Pin the parent with `--subscription` and `--rg`; constrain tenant through the deployment manifest and independently checked account context. Never derive the expected parent from the probe configuration. Every deployment attempt needs fresh after-deployment proof, not current mtime or a reused nonce. Readiness rechecks the full signed envelope/key, bundle and current configuration against previously verified evidence. Changed signatures invalidate it. Required audit uses remote ACK; local overlay fsync is not hosted durability. --- ## Kratos-export adaptation (manifest still required) A **Kratos-exported project** (`src/hosted-agent/` + `use-cases/<x>/`, trimmed `infra/`) can be adapted by the **coding agent**. The Python gate still requires a manifest; recognizing the directory shape does not bypass `_load_manifest`. See [`docs/KRATOS-BRIDGE.md`](../../docs/KRATOS-BRIDGE.md#adapt-before-safe-check). - **Author the deployment contract explicitly.** Inspect compiled Bicep, `azure.yaml` and use-case sources. Use the [`threadlight-design` manifest schema/example](../threadlight-design/SKILL.md#5-specsmanifestjson) to create `specs/manifest.json` with a `deployment_manifest` object and reviewed resource types, selectors, services/roles, integrations, jobs and channels. Run `threadlight-safe-check --phase design --manifest specs/manifest.json` from the export root after installing the supported package. - **There is no `--from-infra` flag.** Missing/invalid manifest or missing `deployment_manifest` still means **exit code 2**. Do not fabricate empty manifests or postdeploy proof. - **Trimmed infra is intentional, not automatically passing.** Do not add APIM/frontend expectations merely because another template has them; retain any actually selected customer network/governance requirement. Derive `required_aca_roles` from deployed Container App resources, not from the presence of a hosted-agent source directory. - **`evals/` absence is expected** pre-backfill (Kratos `_SKIP_DIRS`); it is not a deploy defect. `threadlight-deploy` Kratos-export mode backfills it. After adaptation, run postdeploy with `--manifest specs/manifest.json`, explicit approved `--subscription` / `--rg`, and review `tests/postdeploy-manifest.json`. The gate still fails genuine contract gaps; omission from the export does not waive readiness obligations. --- ## CLI ```bash # From repo root python3 tests/safe_check.py --phase design # after threadlight-design Phase 1-3 python3 tests/safe_check.py --phase pre-deploy # immediately before azd up python3 tests/safe_check.py --phase post-deploy # immediately after azd up returns 0 ``` Exit codes: | Code | Meaning | |---|---| | `0` | Gate passed (gaps empty) | | `1` | Gate failed (gaps non-empty); manifest written with details | | `2` | Missing prerequisite (no `specs/manifest.json`, no `deployment_manifest{}` block, env vars missing) | | `3` | Tooling error (Azure auth, `az` not on PATH, etc.) | Optional flags: ```bash --rg <name> # override AZURE_RESOURCE_GROUP env var (post-deploy) --subscription <id-or-name> # post-deploy account; defaults to current CLI account --manifest <path> # override default specs/manifest.json --out <dir> # override default tests/ output dir --quiet # only print final OK / FAIL line + exit code ``` --- ## Files in this skill ``` threadlight-safe-check/ ├── SKILL.md (this file) ├── scripts/ │ └── safe_check.py (single-file Python module — the CLI) └── tests/ └── test_safe_check.py (integration_binding_gaps + example parity) ``` The legacy CLI remains a copyable file. Selected governance additionally requires the portable `threadlight-governance-safe-check` package; copying only the CLI never silently skips a selected governance gate. Run the shipped tests standalone (no pytest required): ```bash python3 skills/threadlight-safe-check/tests/test_safe_check.py # or, with pytest: python3 -m pytest skills/threadlight-safe-check/tests/ -q ``` `test_safe_check.py` also asserts the example copy shipped as `examples/returns-triage-governed/tests/safe_check.py` stays **byte-for-byte** identical to `scripts/safe_check.py`, so the two never drift. --- ## Phase 1 — `--phase design` (post-design check) **Inputs:** `specs/SPEC.md`, `specs/manifest.json` **Asserts:** 1. `specs/manifest.json` exists and parses as JSON. 2. Top-level `deployment_manifest{}` block present (added by `threadlight-design` Phase 3 — see `threadlight-design/SKILL.md` §3 for the schema). 3. `deployment_manifest.module_selectors` is a `dict[str, "yes"|"no"]` covering every selector named in SPEC § 11c table. 4. `deployment_manifest.services[]` lists every service that needs a container image; every entry has `name`, `host`, `src`. 5. `deployment_manifest.scheduled_jobs[]` listed iff `aca-job: yes`. 6. `deployment_manifest.channels[]` lists every Human Interaction channel from SPEC § 8. 7. `deployment_manifest.expected_resource_types[]` non-empty and contains the canonical `Microsoft.*` type for every `yes` selector per the table in **Selector → resource type map** below. **Common gaps caught:** - SPEC § 11c says `aca-bot: yes` but `module_selectors` doesn't list it (drift between SPEC text and manifest contract) - `services[]` references `src/workspace` but SPEC § 8 has no UI channel (orphan service) - `expected_resource_types[]` missing `Microsoft.BotService/botServices` even though `aca-bot: yes` (selector mapping incomplete) --- ## Phase 2 — `--phase pre-deploy` (pre-`azd up` check) **Inputs:** `specs/manifest.json`, `azure.yaml`, `infra/main.bicep`, `infra/**/*.bicep`, `src/**/Dockerfile` **Asserts:** Three-column matrix per `yes` selector — every column populated: | Selector | `azure.yaml` services | `infra/main.bicep` module ref | `src/<dir>/` Dockerfile | |---|---|---|---| | `aca-mcp` | `name: mcp`, `host: containerapp`, `project: ./src/mcp` | `module mcpApp '...container-app.bicep'` with `serviceName: 'mcp'` | `src/mcp/Dockerfile` + `server.py` | | `aca-bot` | `name: bot`, `host: containerapp`, `project: ./src/bot` | `module botApp '...container-app.bicep'` with `serviceName: 'bot'` **AND** `module botService 'bot/bot-service.bicep'` | `src/bot/Dockerfile` + `bot.py` + `app.py` + `teams_package/manifest.json` | | `aca-job` | `name: <job>`, `host: containerapp`, `project: ./src/jobs/<job>` | `module job 'jobs/aca-job.bicep'` (or equivalent under `infra/jobs/`) | `src/jobs/<job>/Dockerfile` + `main.py` (cron entrypoint) | | `workspace-ui` | `name: workspace`, `host: containerapp`, `project: ./src/workspace` | `module workspaceApp '...container-app.bicep'` with `serviceName: 'workspace'` | `src/workspace/Dockerfile` + ACA-served HTML/SPA. **NOT a static `index.html` only.** | | `foundry-iq-index` | n/a (provisioned by hook) | `module knowledge 'modules/ai-search.bicep'` (the index) | `scripts/postprovision.py` calls `provision_knowledge_base()` | Plus **two orphan checks** (caught orphan `infra/bot/aca.bicep` files left for an entire deploy cycle): 1. **Bicep-module orphan check.** Every `infra/<dir>/*.bicep` (excluding `core/`, `modules/`) must be referenced from `infra/main.bicep` via `module ... '<path>'`. Otherwise → orphan; either wire or delete. 2. **`src/`-folder orphan check.** Every `src/<dir>/` must map to a declared `azure.yaml` service (or be `src/agent/` which has its own host). Otherwise → orphan; either wire or delete. **Common gaps caught:** - `aca-bot: yes` but `azure.yaml` has no `bot` service → silent partial PoC - `infra/bot/aca.bicep` exists but no `module botApp` line in