runner-diagnostics

# Diagnose a Runner failure Trace the failure from a public status comment to bounded local diagnostics. Keep tokens, internal hostnames, absolute paths, raw adapter output, and private deployment details out of issue comments and public pull requests. ## Start from the failing identifier 1. Record the public session ID, job ID, trigger-comment URL, status-comment URL, approximate time, and command (`/new`, `/resume`, or `/cancel`). 2. Locate the scoped Runner directory containing `state.json` and `logs/`. Use the runner's configured `--state` path rather than assuming a repository name or service-account layout. 3. Resolve the job or session through `logs/index.ndjson`, then read only the matching job and session records. ```text state.json logs/ runner.ndjson errors.ndjson index.ndjson jobs/<job-id>.ndjson jobs/<job-id>-acpx-stdout.log jobs/<job-id>-acpx-stderr.log sessions/<public-session-id>/<turn-correlation-id>.ndjson ``` Use `jq` and narrow `rg` queries. Do not dump all logs. ```bash rg -n '"id":"<job-id>"|"id":"<public-session-id>"' logs/index.ndjson rg -n '"level":"error"|"component":"(intake|dispatch|acpx)"' logs/jobs/<job-id>.ndjson tail -100 logs/jobs/<job-id>-acpx-stderr.log ``` ## Reproduce in the correct order 1. Run `issue-spec runner preflight` as the same OS service user. 2. For Codex, add `--verify-agent-runtime` to create one tools-denied ACP session. This validates the effective ACPX adapter and any explicit `--model`; it is not a substitute for the bubblewrap check. 3. Check the durable job's `sandbox.agent_runtime`, `sandbox.bwrap`, `sandbox.env_decisions`, and bounded diagnostics before changing any config. 4. Re-run only the smallest failed layer: signed webhook delivery, command authorization, clone, read-only Git command, documentation-only commit and push, then Provider `change.create` when that capability is configured. ## Diagnose the common boundaries ### Command intake - Confirm the command starts at the beginning of the comment. - Confirm the author is allowed and has repository write-equivalent permission. - For self-hosted intake, confirm the subscription ID, secret rotation window, receiver reachability, and server/runner clocks. - Correlate the delivery ID, `cycle_id`, `trigger_comment_id`, and job ID before deciding that a command was ignored. ### Codex or ACPX - Treat `acpx` and its configured `agents.codex` override as the effective runtime. A host `codex --version` does not prove that the adapter supports a model. - Inspect `codex-acp` preflight and `sandbox.agent_runtime`. The latter stores only `builtin` or a safe adapter description, never the host config path. - When ACPX rejects a model, use the exact adapter-advertised ID, including a reasoning-effort suffix. Remove an unnecessary explicit `--model` only after confirming the service-user default works. - Check npm/npx access or the pre-cached pinned adapter package. Do not paste raw ACPX output into an issue because it can contain environment diagnostics. ### Proxy and network - Inspect the systemd environment file and `proxy_inherited:*` decisions in sandbox metadata. Standard `HTTP_PROXY`, `HTTPS_PROXY`, and `NO_PROXY` names are inherited by the sandbox. - Put directly reachable receiver, issue-server, and code-host addresses in `NO_PROXY`. Keep proxy credentials out of the unit, logs, and issue comments. - Restart the service after editing the environment file, then re-run the live runtime probe as the service user. ### Sandbox, Git, and SSH - Check `sandbox.bwrap` and `sandbox.preflight_result` before considering `--unsafe-no-sandbox`; do not use unsafe mode as a routine workaround. - For brokered credentials, compare the active Source Binding identity and clone URL, then verify lease acquisition and revocation. - For `--allow-host-ssh`, verify the dedicated Runner account's non-interactive SSH access and `known_hosts`. Treat all authority reachable by that identity as available to every job in that Runner boundary. - Do not solve a missing code-host CLI by mounting arbitrary host binaries into bubblewrap. ### PR/MR creation - Distinguish commit/push success from change creation. A job can safely push a branch and still lack authority to create a PR/MR. - If a configured `issue-spec.code-provider/v1` bridge advertises `change.create`, verify its request and resulting change URL. Keep the company-specific wrapper and registry operator-owned. - If the bridge does not advertise `change.create`, report the branch/revision evidence and create the change outside the sandbox. Do not broaden sandbox mounts to bypass this boundary. ## Report safely Report the public session/job identifiers, failing boundary, timestamp, sanitized error category, relevant preflight check, and the smallest safe next action. Link to the status comment or change URL when available. Exclude raw logs, credentials, internal addresses, full filesystem paths, and secret-like environment values.