latitude-dev

Preparing a production release, pushing a vX.Y.Z release tag, running scripts/release.sh, or updating CHANGELOG.md with the changes that are about to be deployed to production.

apps/web UI — routes, @repo/ui, TanStack Start server functions and collections, navigation (Link vs useNavigate), forms (useForm + createFormSubmitHandler + fieldErrorsAsStrings for Zod field errors), Tailwind layout rules, design-system updates, and useEffect / useMountEffect policy.

Multi-channel notifications. Adding a new notification kind, group, or channel; in-app + email delivery; per-user prefs; project-level gates; idempotency.

Review the current conversation context and git changes, then persist durable repository knowledge into `dev-docs/*.md` by domain and into `AGENTS.md` for cross-cutting repo rules. Use after features, fixes, refactors, architecture changes, schema changes, or when the user mentions docs, documentation, design, architecture, business logic, conventions, or `AGENTS.md`.

ClickHouse queries, Goose migrations, chdb test schema, or telemetry storage paths.

Adding or changing routes in `apps/api`. One source of truth (`defineApiEndpoint` + a Zod schema) becomes an HTTP endpoint, an OpenAPI operation, an MCP tool, and a TS SDK method — descriptions and contracts must be written with all four readers in mind.

Drizzle schema, repositories, RLS, SqlClient wiring, Postgres migrations, psql / reset, or platform mappers (toDomain* / toInsertRow).

Queues and workers, domain event publishers, async notifications or projections, or not doing that work inside HTTP handlers.

Use this skill when a developer wants to annotate their LLM outputs, set up an annotation process, or improve existing annotations. Triggers on: "help me annotate my LLM outputs", "review my annotations", "set up an annotation rubric", "my annotations feel inconsistent", "how do I label my traces", "I want to rate my AI outputs", "help me build a scoring rubric", "are my annotations any good", "I need to label good and bad outputs", "improve my annotation process". Two modes: (1) starting from scratch — helps define criteria and build a rubric before annotating; (2) already annotated — reviews existing annotations for vague criteria, missing reasoning, and inconsistency, then helps fix them.

Use this skill when a developer wants to decide what type of evaluation to build for their AI system. Triggers on: "should I use a rule or a judge", "what type of eval should I build", "decide eval type", "judge vs programmatic rule", "LLM-as-judge vs rule-based eval", "which evaluation type should I use", "how do I evaluate [X]", "what eval should I use for this failure", "is this a rule or a judge", "how should I evaluate my AI automatically", "what kind of eval fits this issue". Takes one or more failure modes or quality dimensions and returns a concrete type recommendation — programmatic rule, LLM-as-judge, or composite — with rationale and a suggested implementation path.

Use this skill when a developer wants to check whether their existing evaluations are trustworthy and well-targeted. Triggers on: "audit my evals", "are my evals any good", "review my evaluation setup", "check my LLM judges", "are my evaluations reliable", "something feels off with my eval scores", "inherited an eval system", "my evals are passing but the product feels broken", "post-build eval check", "validate my eval pipeline". Inspects existing eval artifacts — judge prompts, annotation data, issue reports, alignment scores — and produces a prioritized findings report with a concrete fix for each problem. Do NOT use this to build new evals from scratch — use llm-evals-checklist first, then llm-judge-creator.

Use this skill when a developer wants to check if they're ready to build evaluations for their AI system. Triggers on: "am I ready to build evals", "can I start building evaluations", "eval readiness check", "pre-eval checklist", "what do I need before building evals", "how do I know if I'm ready to evaluate my AI", "should I build evals now", "eval prerequisites", "do I have enough to start building evals". Runs a readiness check across four prerequisites — tracing, logs, annotations, issues — before a developer invests time in building evaluations. Checks the codebase and project files first; only asks the developer when it cannot determine the answer itself.

Use this skill when a developer wants to build or expand a golden dataset for regression testing. Triggers on: "build a golden dataset", "create a test dataset", "curate eval examples", "promote logs to dataset", "build regression test cases", "I need a golden dataset", "select good examples for testing", "create a baseline dataset", "curate passing traces". Takes production logs with eval results and helps the developer select, review, and curate representative examples into a structured golden dataset ready for regression testing.

Use this skill when a developer wants to find out what's going wrong with their LLM or AI system in production. Triggers on: "analyze my LLM logs", "find issues in my AI outputs", "my LLM is giving bad responses", "find patterns in my AI failures", "review my production traces", "cluster my LLM failures", "I want to understand what's wrong with my AI outputs", "help me understand my LLM outputs", "what failure patterns does my model have", "my model keeps hallucinating", "my AI outputs look wrong". Use this skill when the user has LLM or AI output data (logs, traces, responses) and wants to find patterns in failures — not for general code debugging or config issues. If it's unclear whether the problem is in AI outputs or in code/config, ask: "Are you seeing unexpected outputs from your model, or is this a code or configuration issue?" Takes a dev from a pile of raw LLM outputs to a structured, prioritized list of named failure patterns they can actually act on.

Use this skill when a developer wants to validate how well their LLM judge aligns with human judgment. Triggers on: "validate my LLM judge", "check if my judge is accurate", "my judge scores don't match human ratings", "calibrate my evaluator", "how reliable is my judge", "measure judge alignment", "test my eval", "check my judge against human labels", "is my judge any good", "validate my evaluator", "my judge is too strict", "my judge keeps missing failures". Takes a judge prompt and human-labeled examples, measures pass agreement rate and failure catch rate, identifies directional bias (too lenient or too strict), and walks the dev through targeted fixes until alignment meets a reliable threshold.

Use this skill when a developer wants to create LLM-as-a-judge evaluators for their AI system. Triggers on: "create an LLM judge", "build an eval for my AI", "automate my evaluations", "create a judge prompt", "build LLM-as-a-judge", "turn my annotations into evals", "create evals from my issue report", "I want to evaluate my AI automatically", "how do I scale my evaluations", "build judges from my traces". Takes an issue report (from llm-issue-discovery) or annotated traces and produces ready-to-use LLM-as-a-judge prompts — one per evaluation dimension — that the dev can copy-paste into their eval setup.

Add or review Latitude Telemetry for LLM apps. Use for Latitude tracing, LLM observability, missing traces, OpenTelemetry/OTLP integration in TypeScript, Python, and other runtimes, including using Latitude MCP to discover telemetry configuration values.

Integrate Latitude telemetry into any backend project for LLM observability. Use when the user wants to add Latitude tracing, logging, or observability to their LLM calls. Covers three approaches based on language and needs -- SDK telemetry wrapper (Python, TypeScript), raw OpenTelemetry OTLP (any language with OTel), or the Create Log API (any language with HTTP). Triggers on tasks involving Latitude telemetry, tracing LLM calls, sending traces to Latitude, or adding LLM observability.

PromptL syntax guide for writing prompts in the Latitude platform. This skill should be used when writing, reviewing, or editing PromptL prompts. Triggers on tasks involving creating prompts, configuring LLM parameters, using variables, conditionals, loops, chains, tools, agents, or any prompt engineering in Latitude.