comet-ml

Authoring Fern MDX documentation pages for the Opik docs site, plus release-note and changelog routing. Use when writing or updating pages under apps/opik-documentation/documentation/fern/, drafting PR descriptions, or picking the right changelog surface.

Use when a developer wants to add, write, or create an end-to-end test for an Opik feature, page, or branch — e.g. "add an e2e test for the experiments comparison page", "write a test for the feature I just built", "e2e test for this branch", "cover the…

Add product analytics (BI) events to Opik features. Use when wiring events on the frontend, the backend, or the Python SDK - all three report through Segment to PostHog.

Use when a developer wants an e2e test covering a change they just made — e.g. "explore this feature", "add a test for my PR", "cover the feature in PR

Feature documentation and release notes patterns. Use when documenting changes, writing PR descriptions, or preparing releases.

Build, update, test, and document Opik SDK integrations (Python & TypeScript). Use when adding a new framework/provider integration under sdks/python/src/opik/integrations or sdks/typescript/src/opik/integrations, updating an existing one, or verifying that…

Validate what a ClickHouse query actually costs before merging it — at production scale through read-only environment access, or on a revived Testcontainers dataset extrapolated to 20k/500k/1M entities. Use when a DAO query changes, when an endpoint is slow,…

Python SDK patterns for Opik. Use when working in sdks/python, on SDK APIs, integrations, or message processing.

Pull 2+ Comet experiments via the Comet MCP server (or `comet_ml.API` as fallback) and render a side-by-side markdown table of metrics and hyperparameters. Highlights the winning value per metric row. Use when the user asks to "compare", "diff", "which run…

Query a single Comet experiment via the Comet MCP server (or `comet_ml.API` as a fallback), fetch its metrics and parameters, and compare against thresholds (inline or from `thresholds.yaml`). Reports pass/fail per metric with a markdown table. Use when the…

Manage a Comet Model Registry version's status and/or tags via `scripts/registry_mutate.py`. Status is a single-value enum (None / Development / Staging / QA / Production by default; workspace-configurable). Tags are a multi-value free-form list. Confirms…

Execute `make train`, capture output, and surface the Comet experiment URL, registered models, logged artifacts, and final metrics. Use when the user asks to "run a training run", "train the model", or "kick off an experiment". Pure shell wrapper — does not…

Diff HEAD against main, group the changes by Keep-a-Changelog type (Added/Changed/Fixed/...), propose one-line entries under [Unreleased] in CHANGELOG.md, then edit the file inline. User reviews the diff and stages it themselves. Use before any commit that…

Audit the Makefile against the tooling actually installed (ruff, pytest, mypy, etc.) and the scripts under src/<module>/, then propose missing targets or fix stale ones. Use when adding a new tool or script that should be invokable via make. Preserves…

Regenerate the auto-generated sections of README.md (project tree, common-tasks table) to reflect the current state of pyproject.toml, the Makefile, and src/. Use when project structure changes, deps change, or those sections drift from reality. Preserves any…

Root-cause a specific Opik trace, or a pattern across traces, and return a grounded explanation. Uses the hosted Opik MCP when it is connected, and falls back to SDK scripting otherwise. Returns the root cause, the evidence spans as clickable Opik UI links,…

Surface the Opik traces worth a developer's attention, ranked by signal — errors, failed tool calls, latency, regressions, and low online-eval scores — plus Diagnostics issues. Reads live/production traces via the SDK (search_traces and agent_insights) and…

Build an LLM evaluation and run it against your app, returning an experiment with scores. Covers datasets, LLM judges, RAG evaluation, synthetic data, error analysis, and validating evaluators against human labels. Use when the user wants to measure or…

Add Opik tracing to an existing app and verify a real trace lands. Installs the Opik package, detects the language and LLM framework, adds the minimum tracing, runs a safe representative path, confirms a trace in Opik, and returns the trace link. Use for…

Reference for the Opik SDK — tracing, span types, framework integrations, threads, and the prompt library (Python, TypeScript, REST). Use for "what span types exist", "how do I flush", "track_openai", "add OpikTracer", "version a prompt". To instrument a repo…

Scaffold a brand-new Opik example in this repo from a canonical template under templates/ — either a full use-case demo (templates/use-case-template/) or a standalone utility script (templates/script-template/). Use this whenever the user wants to create,…

Add new items to an Opik evaluation dataset in this repo. Use when the user says "add a case to the dataset", "add items to the Opik dataset", "extend the eval dataset", "seed the dataset", "add test data for the eval", or names new query/expected-output…

Add new items (plain-English assertions) to an Opik test suite in this repo. Use when the user says "add an assertion", "add a test case to the suite", "extend the eval suite", "add a check to the test suite", "the model should also be tested for X", or…

Run an Opik evaluation for an example in this repo — metrics (evaluate) and/or assertions (run_tests) against a dataset/test suite. Use when the user says "run the eval", "evaluate the example", "run the test suite", "score the model", "check the metrics",…

Run an Opik prompt optimization (opik-optimizer) for an example in this repo, and optionally promote the result to the Prompt Library. Use when the user says "optimize the prompt", "run the optimizer", "improve the prompt", "run Optimization Studio", "tune…

This skill should be used when the user needs to add Opik tracing or integrations to their code, instrument an LLM application, or needs reference for Opik SDK usage (Python, TypeScript, REST API). Use for tasks like "add tracing", "instrument my code", "use…

This skill should be used when the user asks about agent architecture, evaluation, metrics, production monitoring, debugging agents, or best practices for building reliable AI agents. Use for questions like "evaluate my agent", "set up production monitoring",…

Add Opik tracing to an existing codebase. Detects language (Python/TypeScript), identifies LLM frameworks, adds appropriate decorators and integrations, marks entrypoints, and wires up environment config. Use for "instrument my code", "add opik tracing", "add…

Opik observability for LLM agents — Prompt Library, Local Runner (opik connect), Test Suites, threads, integrations. Use for "manage my prompts", "connect my agent", "evaluate my agent" or "integrate with Opik".

Scaffold a brand-new Comet example in this repo from the canonical template under templates/integration-example/. Use whenever the user wants to create, add, or start a new example, demo, or integration on Comet — phrasings like "add a Comet example for X",…

How the opik-cipx cost-tracking proxy works — architecture, the CLI, where it stores state, how it's enabled and disabled, privacy and telemetry, and how to read `opik-cipx status`. Use whenever the user asks about opik-cipx, token/cost tracking, or Opik…