fix-agent-qa-ticket

# Fix a QA ticket (agtqa_*) `agtqa_*` = an Agent Conversation Triage Ticket: a reviewer's flagged comments against specific turns of a real Architect/ConvAI conversation. This skill turns one of those tickets into a verified, branch-scoped fix, entirely through the raw ConvAI REST API (host `https://api.elevenlabs.io`, header `xi-api-key`). It never touches main directly — it stages everything on a branch and leaves merging to the user. **Needs from the user**: an API key and a ticket id (`agtqa_...`) or a URL containing one. Everything else — agent id, conversation id, branch — is derived from the ticket. **Platform-only.** Do not write repo code as part of this skill unless the root cause is actually a code bug (rare — most triage findings are prompt/procedure gaps). If code IS the root cause, say so and hand off instead of guessing at a platform-side workaround. ## 1. Fetch the ticket ```bash curl -s "https://api.elevenlabs.io/v1/convai/conversation-triage-tickets/$TICKET_ID" \ -H "xi-api-key: $API_KEY" ``` Returns `conversation_id`, `agent_id`, `qa_comment`, `ticket_comments[]`, `turn_comments[]` (each with `turn_index` + `comment`), `status` (`open`/`in_progress`/`resolved`). ## 2. Fetch the conversation and read the flagged turns ```bash curl -s "https://api.elevenlabs.io/v1/convai/conversations/$CONVERSATION_ID" \ -H "xi-api-key: $API_KEY" ``` `transcript[]` has per-turn `role`, `message`, `tool_calls` (each with `tool_name` + `params_as_json`). Read a window around every `turn_index` named in `turn_comments` — usually a couple turns before and after — to understand what the agent actually did and said. The reviewer's comments tell you *what's wrong*; the transcript tells you *why*. Don't skip straight to guessing the fix — find the specific tool call, prompt gap, or missing instruction that produced the bad turn. The conversation also carries `branch_id` and `version_id` — useful context, but **do not build your fix branch off this conversation's branch** unless it's demonstrably the right base (check `is_archived` / whether it's actually merged into main — see step 3). It's just where the flagged conversation happened to run. ## 3. Create a fix branch off main's actual tip Get the agent and find its real main branch: ```bash curl -s "https://api.elevenlabs.io/v1/convai/agents/$AGENT_ID" -H "xi-api-key: $API_KEY" # -> .main_branch_id, .branch_id (top-level is usually main) ``` Then fetch that branch to get its true HEAD version (do not assume — a stray personal/archived branch can look tempting but have unrelated commits in its ancestry): ```bash curl -s "https://api.elevenlabs.io/v1/convai/agents/$AGENT_ID/branches/$MAIN_BRANCH_ID" \ -H "xi-api-key: $API_KEY" # -> .most_recent_versions[0].id is main's real tip version_id ``` Create the branch from that exact version: ```bash curl -s -X POST "https://api.elevenlabs.io/v1/convai/agents/$AGENT_ID/branches" \ -H "xi-api-key: $API_KEY" -H "Content-Type: application/json" \ -d '{"name": "<you>/qa-<ticket-suffix>-<short-desc>", "description": "Fix for '"$TICKET_ID"'", "parent_version_id": "<main-tip-version-id>"}' ``` Required fields are `name`, `description`, `parent_version_id` (all three, or you get a 422 listing what's missing). Response: `{created_branch_id, created_version_id}`. ## 4. Find and fix the root cause Usually one of: - **A procedure** missing an instruction (list via `GET .../agents/{id}/branches/{b}/procedures`, fetch full content via `GET .../procedures/{pid}`). Search procedure names/triggers for the relevant topic (dashboards, refunds, whatever the ticket concerns). - **The system prompt** (`conversation_config.agent.prompt.prompt` off `GET /v1/convai/agents/{id}?branch_id={b}`). ### Editing an existing procedure (draft + publish dance) Editing an existing procedure is **two steps**, not one — there is no direct commit-on-PATCH for procedures: ```bash # 1. Stage the draft (full content: frontmatter + body, or per-field if the # procedure isn't markdown-frontmatter style) curl -s -X PATCH \ "https://api.elevenlabs.io/v1/convai/agents/$AGENT_ID/branches/$BRANCH_ID/procedures/$PROCEDURE_ID/draft" \ -H "xi-api-key: $API_KEY" -H "Content-Type: application/json" \ -d '{"name": "...", "content": "...", "type": "free_form", "trigger": "..."}' # 2. Publish ALL pending procedure drafts on the branch by re-committing the # agent with any partial merge body (re-sending the branch's current, # unmodified prompt is the simplest no-op payload). Fetch the CURRENT prompt # scoped to $BRANCH_ID first (not main's) so you don't clobber other # branch-local differences: curl -s "https://api.elevenlabs.io/v1/convai/agents/$AGENT_ID?branch_id=$BRANCH_ID" -H "xi-api-key: $API_KEY" \ | jq -r '.conversation_config.agent.prompt.prompt' > current_prompt.txt curl -s -X PATCH "https://api.elevenlabs.io/v1/convai/agents/$AGENT_ID?branch_id=$BRANCH_ID&version_description=..." \ -H "xi-api-key: $API_KEY" -H "Content-Type: application/json" \ -d "{\"conversation_config\": {\"agent\": {\"prompt\": {\"prompt\": $(jq -Rs . < current_prompt.txt)}}}}" ``` Verify by re-`GET`ting the procedure and confirming the `version_id` changed and the new content is present. A direct `PATCH .../procedures/{pid}` (no `/draft` suffix) does not exist — returns 405. Creating a brand-new procedure via `POST .../procedures` commits immediately, but don't use that to "replace" an existing one in place — it leaves two procedures with overlapping/duplicate triggers, which is worse than the bug you're fixing. ### Editing the prompt / criteria directly No draft dance needed — a partial-merge `PATCH /v1/convai/agents/{id}?branch_id={b}` commits straight to branch HEAD and returns a new `version_id`: - prompt: `{"conversation_config":{"agent":{"prompt":{"prompt":"..."}}}}` - criteria: `{"platform_settings":{"evaluation":{"criteria":[...]}}}` (send the full array; each `conversation_goal_prompt` max 2000 chars) ## 5. Add a simulation test that reproduces the ticket's scenario **Always use a `simulation` test for Architect/DOM agents** — `llm`/`response` tests only grade the agent's first action, which for Architect is almost always `start_procedure`, so the judge returns useless `unknown`/`failure` verdicts. ```bash # Optional: group tests for this ticket curl -s -X POST "https://api.elevenlabs.io/v1/convai/agent-testing/folders" \ -H "xi-api-key: $API_KEY" -H "Content-Type: application/json" \ -d '{"name": "QA '"$TICKET_ID"'"}' curl -s -X POST "https://api.elevenlabs.io/v1/convai/agent-testing/create" \ -H "xi-api-key: $API_KEY" -H "Content-Type: application/json" \ -d '{ "type": "simulation", "name": "<short description> ('"$TICKET_ID"')", "parent_folder_id": "<folder id or omit>", "simulation_scenario": "<persona + exactly what they do, mirroring the ticket>", "success_conditions": ["<checklist item 1>", "<checklist item 2>", ...], "simulation_max_turns": 8, "tool_mock_config": {"mocking_strategy": "all", "fallback_strategy": "raise_error"}, "chat_history": [{"role": "user", "message": "...", "time_in_call_secs": 0}], "dynamic_variables": {"tier": "enterprise", "product": "conversational_ai", "objective": "...", "userInfo": "{}", "chatHistory": "[]"} }' ``` Gotchas that produce 422s: - Every `chat_history` entry needs `time_in_call_secs` (even `0`), or you get a `missing` error pointing at that field. - `chat_history` must end with a user message. - A `system`-type `tool_result` inside `chat_history` needs `is_error` + `tool_has_been_called` fields. - `dynamic_variables` should include whatever the prompt templates on (tier/product/objective/userInfo/chatHistory at minimum) or templating crashes mid-run. - `mocking_strategy` is an enum of exactly `all` / `selected` / `none` — there is no fourth value. **`"all"` ignores `tool_mock_overrides` entirely** — it mocks every tool call with the generic fallback error, even if you've set a per-tool `mock_result`. If the fix you're testing depends on a specific tool actually returning realistic data (e.g. `list_branches` returning a branch list so the agent can resolve a name to an id), you MUST use `mocking_strategy: "selected"` with `mocked_tool_ids: [...]` naming every tool you want mocked — `"all"` silently no-ops your overrides and you'll see "no mock matched" in the transcript even though the override is saved correctly on the test (verify via `GET /v1/convai/agent-testing/{test_id}` if a mock isn't taking effect — the stored config can look right while the run still fails, which means the strategy, not the override, is wrong). - `tool_mock_overrides` shape: `{"<tool_name>": [{"mock_result": "<json-string>", "parameter_conditions": [], "is_error": false}]}`. `mock_result` is a JSON-encoded **string**, not a nested object — build it with `json.dumps(...)` (or equivalent) before embedding it in the request body. An empty `parameter_conditions` list means "match unconditionally." - `mocking_strategy: "all"` + `fallback_strategy: "raise_error"` (no overrides) makes *every* tool call error mid-conversation ("technical difficulties"). Fine only for testing something that doesn't depend on a tool succeeding (e.g. a confirmation-message wording fix where the tool call itself is incidental); if the flow needs a tool to actually succeed with realistic data, use `selected` + overrides instead of reaching for `call_real_tool`. - DOM-write tools (navigate, set_dashboard_filters, etc.) still may not fully green in the sim harness even with `selected` mocking if you haven't mocked every tool in the chain — treat persistent failures there as behavior documentation, not a bug, once you've confirmed the mocking strategy itself isn't the culprit. Write the `success_conditions` directly against the reviewer's `turn_comments` — each comment should map to a checklist item the grader can verify. Word each condition to name the CORRECT id/value explicitly (e.g. "uses id X, not the raw name string and not id Y") rather than just "doesn't guess" — a vague condition lets a new, differently-wrong failure mode (e.g. passing the name itself as the id) slip through as a pass. **Editing a sim test = delete + recreate.** There is no in-place edit for a simulation test (`PUT` on a test id is response/llm-only and rejects `type: simulation`). If a run reveals your mock config was wrong, `DELETE /v1/convai/agent-testing/{test_id}`, fix the payload, and `POST .../create` again — then re-attach and re-run. Attach it to the agent/branch, then run it: ```bash curl -s -X POST "https://api.elevenlabs.io/v1/convai/agents/$AGENT_ID/testing/attach-test" \ -H "xi-api-key: $API_KEY" -H "Content-Type: application/json" \ -d '{"test_id": "'"$TEST_ID"'", "branch_id": "'"$BRANCH_ID"'"}' curl -s -X POST "https://api.elevenlabs.io/v1/convai/agents/$AGENT_ID/run-tests" \ -H "xi-api-key: $API_KEY" -H "Content-Type: application/json" \ -d '{"tests": [{"test_id": "'"$TEST_ID"'"}], "branch_id": "'"$BRANCH_ID"'", "repeat_count": 1}' # -> {id: suite_id, test_runs: [{test_run_id, status: "pending", ...}]} ``` ## 6. Poll for the result Sim runs take a few minutes. Poll `GET /v1/convai/test-invocations/{suite_id}` until `test_runs[0].status` leaves `pending`. Use a background poll loop (Bash `run_in_background` or Monitor), not a foreground sleep — do not busy-wait in the conversation. Once terminal, read `test_runs[0].condition_result.result` (`success`/`failure`) and `.rationale.messages` (per-criterion grader reasoning) to confirm the fix actually produces the intended behavior — don't just check the pass/fail bit, skim the rationale for whether it's testing what you think it's testing. If it fails: re-read the rationale against the actual procedure/prompt change, adjust, and re-run. Don't loosen `success_conditions` to force a pass unless the condition itself was wrong (too strict/loose) — a passed test the reviewer's concern doesn't check is worse than an honest failure. ## 7. Comment on the ticket