twilio-extract

# Twilio Extract Read-side tool: pulls data OUT of Twilio (conversations, messages, participants, Studio Flow executions/steps, debugger alerts) via the REST API. Also exposes two opt-in mutating commands (`update-attributes`, `close`) gated behind `--yes`, for the same debugging use cases documented in this org's Twilio integration docs. ## When to Activate - "busca las conversaciones de Twilio pa este número" - "qué attributes tiene esta conversación de Twilio" - "revisa la ejecución del Studio Flow pa este execution sid" - "hay alertas recientes en el debugger de Twilio" - Cross-referencing a Twilio conversation with our own DB / Langfuse session ## Credentials — `.env` convention One `.env` file per account/workspace at `~/.twilio_<workspace>.env` — same pattern already in use in this environment (`~/.twilio_qa.env`, `~/.twilio_qa_prod.env`), never committed to git. Template: `.env.example` in this skill's directory. Required vars: ``` WORKSPACE_NAME= # label for logs ACCOUNT_SID= # AC... AUTH_TOKEN= # live Auth Token (Test SID/Token don't work for Conversations/Studio) ``` Var names are bare `ACCOUNT_SID`/`AUTH_TOKEN` (not `TWILIO_*`) to match the existing files. The `export ` prefix is optional — `python-dotenv` strips it. Auth is HTTP Basic: username=Account SID, password=Auth Token. Set `chmod 600` on the env file. ## Tools in this skill - `scripts/twilio_client.py` — reusable `TwilioClient` class (auth, cursor pagination via `meta.next_page_url`, retry on 429/5xx). Import this if writing a custom script. - `scripts/extract.py` — CLI wrapper. Covers the common cases without writing code. ### CLI usage ```bash cd ~/.claude/skills/twilio-extract/scripts uv run extract.py <kind> --env-file ~/.twilio_qa_prod.env [options] ``` `<kind>`: `conversations-by-address` | `conversation` | `messages` | `participants` | `webhooks` | `update-attributes` | `close` | `studio-flow` | `studio-executions` | `studio-steps` | `debugger-alerts` | `health` ```bash # Every conversation a WhatsApp number has participated in (most common query) uv run extract.py conversations-by-address --env-file ~/.twilio_qa_prod.env \ --address "whatsapp:+584149190631" # Single conversation (state, attributes, timers) uv run extract.py conversation --env-file ~/.twilio_qa_prod.env --sid CH68bdf4bfe0d3404e9ef707220eb78f7e # Messages in a conversation, oldest first uv run extract.py messages --env-file ~/.twilio_qa_prod.env \ --sid CH68bdf4bfe0d3404e9ef707220eb78f7e --order asc # Scoped webhooks attached to a conversation (how our adapter gets onMessageAdded) uv run extract.py webhooks --env-file ~/.twilio_qa_prod.env --sid CH68bdf4bfe0d3404e9ef707220eb78f7e # Studio Flow: recent executions, then the steps of one execution uv run extract.py studio-executions --env-file ~/.twilio_qa.env --flow-sid FW17995da90888cd3be079dd26dc04d820 uv run extract.py studio-steps --env-file ~/.twilio_qa.env \ --flow-sid FW17995da90888cd3be079dd26dc04d820 --execution-sid FNxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx # Debugger alerts (errors/warnings Twilio raised recently) uv run extract.py debugger-alerts --env-file ~/.twilio_qa_prod.env # Sanity check credentials before a real pull uv run extract.py health --env-file ~/.twilio_qa_prod.env # Mutating (opt-in, requires --yes): force-close to make the next inbound re-run the Flow uv run extract.py close --env-file ~/.twilio_qa.env --sid CHxxxx --yes # Mutating (opt-in, requires --yes): set attributes — REPLACES the whole blob, # so read `conversation` first and merge client-side if you need to keep existing keys uv run extract.py update-attributes --env-file ~/.twilio_qa_prod.env --sid CHxxxx \ --json '{"langfuse_session_url":"https://us.cloud.langfuse.com/project/.../sessions/..."}' --yes ``` Output: `--output/-o file --format/-f json|jsonl|csv` (default: prints JSON to stdout). `csv` flattens nested fields (attributes/messaging binding/etc.) to JSON strings per cell. `--max-items` caps total items across pages for list calls (`conversations-by-address`, `messages`, `participants`, `studio-executions`, `studio-steps`, `debugger-alerts`) — Twilio's list resources don't return a total count up front, so an unscoped pull can page for a long time on a busy account; the cap is a safety net, not a substitute for narrowing the query (there's no server-side date filter on `ParticipantConversations`, so narrow by address instead). ## API reference (what the client wraps) | Resource | Base | Endpoint | Notes | |---|---|---|---| | Participant Conversations | `conversations.twilio.com/v1` | `GET /ParticipantConversations?Address=...` | The one way to find conversations by phone/WhatsApp number — no reverse lookup by contact name | | Conversation | `conversations.twilio.com/v1` | `GET/POST /Conversations/{sid}` | POST replaces `Attributes`/`State` whole, no server-side merge | | Messages | `conversations.twilio.com/v1` | `GET /Conversations/{sid}/Messages` | `Order=asc\|desc` | | Participants | `conversations.twilio.com/v1` | `GET /Conversations/{sid}/Participants` | Includes `messaging_binding.address`/`proxy_address` | | Conversation Webhooks | `conversations.twilio.com/v1` | `GET /Conversations/{sid}/Webhooks` | Scoped per-conversation webhooks (how a Studio Flow or our adapter gets notified) | | Studio Flow | `studio.twilio.com/v2` | `GET /Flows/{sid}` | Includes the published flow definition | | Studio Executions | `studio.twilio.com/v2` | `GET /Flows/{sid}/Executions` | One per conversation run through the flow | | Studio Steps | `studio.twilio.com/v2` | `GET /Flows/{sid}/Executions/{sid}/Steps` | Which widget/branch fired, in order — the debugging payoff | | Debugger Alerts | `monitor.twilio.com/v1` | `GET /Alerts` | Twilio's own error/warning log (401s, webhook failures, etc.) | | Account (health check) | `api.twilio.com/2010-04-01` | `GET /Accounts/{sid}.json` | Cheapest call to confirm ACCOUNT_SID/AUTH_TOKEN are valid | Full reference: https://www.twilio.com/docs/conversations/api, https://www.twilio.com/docs/studio/rest-api ### Gotchas - **Address format matters.** `ParticipantConversations?Address=` needs the full channel-prefixed address — `whatsapp:+584149190631` for WhatsApp, bare `+584149190631` only for plain SMS/voice. Forgetting the `whatsapp:` prefix returns an empty list, not an error — looks like "no conversations" when really it's a format mismatch. - **Attributes/State writes REPLACE, they don't merge.** `POST /Conversations/{sid}` with `Attributes=...` overwrites the whole JSON blob server-side. To add one key without losing others: `conversation` (GET) first, merge client-side, then `update-attributes` with the full merged JSON. Same contract the app's own `AsyncTwilioClient.update_conversation` follows (`backend/cheo/channels/twilio/client.py`). - **Test SID + Test Token don't work here.** They only simulate REST responses for Conversations API and webhooks — always use the live Account SID + Auth Token. - **Studio Flow executions ≠ conversations.** An execution is one run of the flow triggered by an inbound message; `studio-steps` shows which widget matched (e.g. a whitelist `split-based-on` widget routing to a bot vs. a human queue) — the fastest way to answer "why didn't this number get routed correctly." - **No server-side date filter on `ParticipantConversations`.** It returns everything for that address, oldest first typically unbounded — use `--max-items` and inspect `conversation_date_created` client-side if you only care about a recent window. - **Rate limits / retries.** `TwilioClient` retries 429 (honoring `Retry-After`) and 5xx with backoff, same as the Langfuse extractor. Don't lower retries for bulk pulls across many addresses. ## Cross-referencing with ai-platform's own data (Langfuse / Postgres) This org's Twilio channel (`backend/cheo/channels/twilio/`) links every Twilio conversation to an internal `Conversation` row: - `Conversation.external_id` (Postgres) = the Twilio Conversation SID (`CH...`) - `Conversation.id` (Postgres, our internal UUID) = the Langfuse **session_id** (`build_session_url` in `cheo/integrations/langfuse/client.py` builds the deep-link from `str(conversation.id)` directly — no separate mapping table) So to go from a phone number to its Langfuse trace: 1. `conversations-by-address --address whatsapp:+<number>` -> get the Twilio SID(s) 2. If the conversation is recent, its `conversation_attributes` may already carry `langfuse_session_url` (set by `twilio/adapter.py` on turn completion) — done. 3. Otherwise, look up `Conversation.id` in Postgres by `external_id IN (<SIDs>)` and build the URL yourself: `{LANGFUSE_BASE_URL}/project/{LANGFUSE_PROJECT_ID}/sessions/{id}`. Langfuse tracing runs for every conversation regardless of whether the attribute got persisted — the attribute is a convenience deep-link, not a gate on the session existing. 4. A Twilio conversation with **zero** matching Postgres row usually means it was never routed to Cheo at all (e.g. the provider's Studio Flow whitelist didn't match that number) — not a bug on our side to chase in the DB. Use the `langfuse-extract` skill's `session`/`debug-session` kinds once you have the internal id, to pull the actual trace content. ## Reference Skills - Pulling the Langfuse session/traces once you have the internal conversation id → skill: `langfuse-extract` - Adding/changing Twilio channel logic in the app → not this skill, it's a coding task