runtime-evidence

name	runtime-evidence
description	Use when a runtime claim is unable-to-check statically and needs a dynamic observation: a bounded reviewed command ID, or an approved browser observation.
license	MIT
compatibility	The bundled runner requires Python 3 on a POSIX platform and refuses Windows before execution. The optional-tool path ships no runner and has no such requirement; it needs an approved, host-exposed observation tool.
metadata	{"author":"Mark Beacom","version":"0.1.0"}
allowed-tools	Read Grep Glob Bash

Runtime Evidence

Collect dynamic evidence only after static verification reaches unable-to-check for a runtime claim. Keep verdict assignment in verify; produce a controlled observation package that lets the verifier confirm, refute, qualify, or retain that verdict.

Run the deterministic wrapper only on POSIX platforms. On Windows it returns a structured refusal before reading the allowlist or spawning a command. That constraint belongs to the runner, not the plugin: the optional-tool path ships no runner and depends only on an approved, host-exposed tool, so it remains available where the wrapper is not.

Treat execution as an escalation, not a default search technique. Prefer code, config, tests, and other static primary evidence when those sources settle the claim.

Evidence flow

1. State one atomic runtime claim and preserve the static verification result that explains why execution is necessary.
2. Identify the environment whose behavior matters. Record a concise label, and the explicit absolute working directory when a command will run.
3. Inspect the user-owned allowlist config when one is available, and select an exact existing command ID whose reviewed argv reproduces the claim.
4. When a reviewed command ID matches, take the runner path (steps 5-6). When no config exists or none of its IDs matches, never invent an ID, alter argv in memory, edit the config, or substitute direct shell execution — either take the optional-tool path (step 7) or stop and report the missing reviewed capability.
5. Invoke scripts/run-evidence-command.py with the config, command ID, claim, environment label, cwd, artifact directory, and unique run ID.
6. Preserve the runner's exit status and JSON report. Treat timeout, output-limit termination, spawn failure, and child nonzero exit as observations rather than smoothing them into success.
7. Take the optional-tool path only when every condition in references/optional-tools.md holds. Run it in the main agent, never in the runtime-investigator subagent: a subagent carries a fixed tool grant, so a browser, debugger, or container tool the host exposes is not reachable inside it. Record the optional-tool field set in the output contract below.
8. Return the report to verify for the verdict. Do not create a parallel verdict taxonomy or a bespoke Plan -> Execute -> Verify -> Synthesize workflow.

Two collection paths

	Runner path	Optional-tool path
When	a reviewed command ID reproduces the claim	no reviewed command can represent it
Runs in	runtime-investigator subagent	the main agent
Bounded by	the runner: allowlist, timeout, and output cap	operator approval plus host policy; nothing the plugin enforces
Observation source	command-id=<allowlist key>	tool=<approved tool>@<target>
Artifacts	runner-written report, stdout, stderr	at least one durable artifact, or the claim stays unable-to-check

This plugin ships a runner for the first path only. It supplies no collection mechanism for the second — just the approval conditions and the recording contract. If the host exposes no suitable tool, that path is unavailable and the claim stays unsettled.

Sanctioned execution path

Use the deterministic wrapper as the default path:

python3 "${CONTEXT_KIT_RUNTIME_EVIDENCE_ROOT}/scripts/run-evidence-command.py" \
  --config "${CONTEXT_KIT_RUNTIME_EVIDENCE_CONFIG}" \
  --command-id "<reviewed-id>" \
  --claim "<atomic runtime claim>" \
  --environment-label "<environment>" \
  --cwd "<absolute working directory>" \
  --artifact-dir "${CONTEXT_KIT_DATA}/runtime-evidence" \
  --run-id "<unique-run-id>"

Inside Claude Code plugin components, use ${CLAUDE_PLUGIN_ROOT}/scripts/run-evidence-command.py when the neutral plugin root variable is not available. Prefer CONTEXT_KIT_* variables in portable instructions.

The wrapper passes configured argv directly to the operating system without a shell. It enforces a config-defined timeout and per-stream output cap, refuses unknown command IDs, refuses unsafe config ownership/permissions where POSIX metadata is available, and requires explicit locations instead of guessing.

Safety rules

• Require commands to be supplied and reviewed by the user or repository owner before execution.
• Treat the allowlist as a command-selection boundary, not a side-effect proof. An allowlisted executable can still mutate state, access credentials, use the network, or spawn descendants.
• Keep host-level Bash permission separate from wrapper policy. A host may allow broad Bash while the wrapper permits one command ID, or may block the wrapper despite a valid config.
• Never reinterpret command strings, append flags, interpolate claim text into argv, or invoke sh, bash, eval, or another command as a fallback.
• Never hide failure. Preserve wrapper and child exit codes and include limitations and cleanup status in the handoff.

Optional runtime tools

Use a browser, debugger, container inspector, or host-specific runtime tool only when the user has approved that observation path and the host exposes the tool. This is the sanctioned branch when no reviewed command ID can represent the claim, and it runs in the main agent — see the path table above. It records the optional-tool field set with tool=<approved tool>@<target> as the observation source. When no suitable tool is exposed, report the missing capability and leave the claim unsettled. Do not replace it with a newly invented command, and do not route it to the runtime-investigator subagent, whose fixed tool grant cannot reach a host-exposed observation tool.

Output contract

Return every field below for every attempted collection. Each field states its own runner-path and optional-tool-path definition where they differ; use the definition for the path you took rather than assuming the other path's values apply.

• Claim — the atomic runtime statement under test.
• Observation source — command-id=<exact allowlist key> on the runner path, never a reconstructed command string; tool=<approved tool>@<target> on the optional-tool path.
• Environment — runner path: label, cwd, platform, and interpreter metadata from the runner report. Optional-tool path: label plus the target identity the tool actually observed — URL/origin, container or process, device/viewport assumptions, and build or deploy identifier when known.
• Observations — runner path: exit code, termination reason, bounded stdout/stderr excerpts, byte counts, and truncation flags. Optional-tool path: the user-visible or instrument-visible state observed, plus console, network, or log findings when relevant. State the observation window and write not-applicable for process-exit fields rather than inventing an exit code.
• Artifact/output pointers — runner path: report, stdout, stderr, and config digest/path. Optional-tool path: at least one durable artifact the tool produced (screenshot, trace, HAR, recording, log export). Capture one before ending the interaction; an observation nobody else can inspect is not evidence, and verify's observation form requires a pointer. When nothing durable was retained, report the observation as not-collected and leave the claim unable-to-check with evidence (none), naming the attempted source and the artifact that would need to be captured. There is no config digest on this path; write not-applicable.
• Verdict-ready evidence — concise facts suitable for the verify verdict taxonomy without assigning a new taxonomy here.
• Limitations — side-effect uncertainty, missing tools, environment gaps, output truncation, timeout, or other constraints. On the optional-tool path, state that no allowlist bounded the interaction and name what was not observed.
• Cleanup status — runner path: whether no cleanup was needed or the process group was terminated. Optional-tool path: session, tab, container, or fixture state left behind. Never imply side effects were reversed on either path.

Read references/evidence-report.md before formatting a handoff.

Composition

Use verify first and last: static verification identifies the runtime gap, and its verifier consumes the resulting evidence. Apply retrieval-strategy through that dependency when locating static evidence is difficult.

Optionally use plan-execute to partition many independent claims before collection. Keep every execution unit bound to an existing allowlist command ID; orchestration never expands execution authority.

References

• references/runner-contract.md - Strict config schema, invocation, ownership checks, artifacts, and exit codes.
• references/evidence-report.md - Required verdict-ready handoff shape.
• references/optional-tools.md - Graceful degradation for browser and host-specific runtime tools.