openmates-realistic-long-chat-validation

## Overview Use this skill when the user wants a realistic long-chat validation run, not a synthetic smoke test. The workflow creates and continues a real OpenMates CLI chat against `https://api.dev.openmates.org`, inspects every turn, validates at least two compression checkpoints, and only then converts the successful share link into a public example chat. This skill intentionally costs real credits and can take a long time. There is no max credit, token, or time cap unless the user adds one for the session. ## Product Contract Source of truth: `docs/plans/realistic-long-chat-cli-validation/plan.yml`. Do not run this workflow as a hidden unattended deterministic script. It is fine to use short helper commands for inspection, checkpoint queries, or CLI command wrapping, but the chat itself must proceed as individual Codex-controlled CLI turns with response review before the next turn. ## Topic Agreement Gate Before starting the CLI chat, ask the user to choose or confirm the long-running topic. If no topic is supplied, suggest examples such as: - Build a company from zero to launch: strategy, positioning, product, finance, hiring, operations, sales, marketing, documents, spreadsheets, and code tools. - Plan a complex nonprofit program: grant strategy, stakeholder maps, budget, outreach, impact measurement, documents, sheets, and dashboards. - Design and launch a technical product: requirements, architecture, GTM, pricing, onboarding docs, analytics sheets, and prototype code. For the current default, use the company-building advisory chat unless the user chooses another topic. ## Model Requirements - Keep the normal chat on Gemini models. - Prefer `google/gemini-3.5-flash-lite` as the primary model where the CLI or mention syntax can select it. - Use the CLI-supported model mention when needed: `@Gemini-3.5-Flash-Lite`. - Before counting the run as valid, verify backend logs show Gemini routing for normal turns and that compression uses the intended cheap Gemini Flash or Flash-Lite path. - If a non-Gemini route is used unexpectedly, pause and ask whether to debug or continue. ## Stop And Pause Conditions Pause immediately and report the issue if any of these occur: - The CLI command fails or the assistant response is missing/unusable. - A generated embed is clearly broken, missing, irrelevant, internally inconsistent, or nonsensical. - Compression appears to complete but no durable checkpoint is persisted. - The model route violates the Gemini requirement. - The chat starts producing content that would be unsuitable for a public example. When pausing, include the chat id, turn number, model when known, affected embed or response, and a clear question: debug/fix now, continue anyway, or abandon and start a better run. ## Live Chat Workflow 1. Ensure the CLI test account is logged in against `https://api.dev.openmates.org`. 2. Confirm the compression model guardrail from the spec is already green. 3. Start one real CLI chat with a natural first prompt for the approved topic. 4. Keep each message as an individual turn. Do not pack the whole transcript into one giant prompt. 5. Progress through many realistic subtopics. For the company-building default, use areas like strategy, customer discovery, positioning, product roadmap, pricing, finance, hiring, operations, legal/privacy, marketing, sales, documents, sheets, code/tooling, and launch review. 6. Ask for generated artifacts during the run, including documents, spreadsheet style planning, and code/tooling where naturally useful. 7. Inspect each assistant response and CLI-rendered embed before continuing. 8. After enough turns, check for the first compression checkpoint. 9. Continue the same chat with more inspected turns after the first checkpoint. 10. Check for and verify the second compression checkpoint. 11. Record sanitized evidence in the spec: chat id, checkpoint ids, `compressed_message_count`, `summary_token_estimate`, observed models, rough credit spend, and pause decisions. ## Checkpoint Verification Use a direct DB or approved API check against the dev stack. A successful live run must show at least two `chat_compression_checkpoints` rows for the same chat. The evidence must prove the run compressed many individual messages across the conversation. A single giant old message does not satisfy this skill. ## Publication Workflow After the live chat succeeds: 1. Create a share link for the real chat. 2. Keep the share `#key=` fragment out of committed files and final public logs. 3. Use the existing `add-example-chat` workflow and `scripts/create-example-chat-from-share.mjs` to scaffold the example. 4. Verify the scaffolded data contains enough history for older-message UX. 5. Deploy before final verification. 6. Verify the deployed example page opens and `Show older messages` is visible and works. 7. Only then give the user the public example URL. ## Evidence Requirements Update `docs/plans/realistic-long-chat-cli-validation/plan.yml` as work proceeds. Record command, run id or manual evidence id, timestamp, subject commit, and a short summary for each verification gate. If a bug pauses the workflow, record it as an attempt/blocker with the next action.