// Investigate stuck runs and execution failures by tracing Symphony and Codex logs with issue/session identifiers; use when runs stall, retry repeatedly, or fail unexpectedly.
| name | debug |
| description | Investigate stuck runs and execution failures by tracing Symphony and Codex logs with issue/session identifiers; use when runs stall, retry repeatedly, or fail unexpectedly. |
log/symphony.log
SymphonyElixir.LogFile (log/symphony.log).log/symphony.log*
issue_identifier: human ticket key (example: MT-625)issue_id: Linear UUID (stable internal ID)session_id: Codex thread-turn pair (<thread_id>-<turn_id>)elixir/docs/logging.md requires these fields for issue/session lifecycle logs. Use
them as your join keys during debugging.
issue_identifier first).session_id from matching lines.session_id across start, stream, completion/failure, and stall
handling logs.# 1) Narrow by ticket key (fastest entry point)
rg -n "issue_identifier=MT-625" log/symphony.log*
# 2) If needed, narrow by Linear UUID
rg -n "issue_id=<linear-uuid>" log/symphony.log*
# 3) Pull session IDs seen for that ticket
rg -o "session_id=[^ ;]+" log/symphony.log* | sort -u
# 4) Trace one session end-to-end
rg -n "session_id=<thread>-<turn>" log/symphony.log*
# 5) Focus on stuck/retry signals
rg -n "Issue stalled|scheduling retry|turn_timeout|turn_failed|Codex session failed|Codex session ended with error" log/symphony.log*
issue_identifier=<KEY>.issue_id=<UUID>.Codex session started ... session_id=....Codex session completed, ended with error, or worker exit
lines.Issue stalled ... restarting with backoff.Codex session failed ....turn_failed, turn_cancelled, turn_timeout, or
ended with error.Agent task exited ... reason=....issue_identifier, issue_id, and
session_id.In Symphony, Codex session diagnostics are emitted into log/symphony.log and
keyed by session_id. Read them as a lifecycle:
Codex session started ... session_id=...session_idCodex session completed ..., orCodex session ended with error ..., orIssue stalled ... restarting with backoffFor one specific session investigation, keep the trace narrow:
session_id for the ticket.rg -n "session_id=<thread>-<turn>" log/symphony.log*Codex session failed ...).turn_* / ended with error).Issue stalled ... restarting with backoff).issue_identifier and issue_id from nearby lines to
confirm you are not mixing concurrent retries.Always pair session findings with issue_identifier/issue_id to avoid mixing
concurrent runs.
rg over grep for speed on large logs.log/symphony.log*) before concluding data is missing.elixir/docs/logging.md conventions.Create a well-formed git commit from current changes using session history for rationale and summary; use when asked to commit, prepare a commit message, or finalize staged work.
Land a PR by monitoring conflicts, resolving them, waiting for checks, and squash-merging when green; use when asked to land, merge, or shepherd a PR to completion.
Use Symphony's `linear_graphql` client tool for raw Linear GraphQL operations such as comment editing and upload flows.
Pull latest origin/main into the current local branch and resolve merge conflicts (aka update-branch). Use when Codex needs to sync a feature branch with origin, perform a merge-based update (not rebase), and guide conflict resolution best practices.
Push current branch changes to origin and create or update the corresponding pull request; use when asked to push, publish updates, or create pull request.