comet-ml

Java backend patterns for Opik. Use when working in apps/opik-backend, designing APIs, database operations, or services.

Feature documentation and release notes patterns. Use when documenting changes, writing PR descriptions, or preparing releases.

Authoring Fern MDX documentation pages for the Opik docs site, plus release-note and changelog routing. Use when writing or updating pages under apps/opik-documentation/documentation/fern/, drafting PR descriptions, or picking the right changelog surface.

Use when a developer wants to add, write, or create an end-to-end test for an Opik feature, page, or branch — e.g. "add an e2e test for the experiments comparison page", "write a test for the feature I just built", "e2e test for this branch", "cover the…

Add product analytics (BI) events to Opik features. Use when wiring events on the frontend, the backend, or the Python SDK - all three report through Segment to PostHog.

Use when a developer wants an e2e test covering a change they just made — e.g. "explore this feature", "add a test for my PR", "cover the feature in PR

Build, update, test, and document Opik SDK integrations (Python & TypeScript). Use when adding a new framework/provider integration under sdks/python/src/opik/integrations or sdks/typescript/src/opik/integrations, updating an existing one, or verifying that…

Validate what a ClickHouse query actually costs before merging it — at production scale through read-only environment access, or on a revived Testcontainers dataset extrapolated to 20k/500k/1M entities. Use when a DAO query changes, when an endpoint is slow,…

Run a candidate against the baseline over an Opik test suite and read the numbers back — which cases broke, which got fixed, the per-metric deltas, worst rows, and whether the two runs are comparable — with the Opik compare-view link. Runs via the SDK; reads…

Surface the Opik traces worth a developer's attention, ranked by signal — Diagnostics issues first, then errors, failed tool calls, latency, regressions, and low online-eval scores. With the Opik MCP connected it lists the project's agent_insights_issue…

Build an LLM evaluation and run it against the app, returning an Opik experiment with scores and its link. Picks a test suite with judge assertions or a dataset with metrics, sources cases from traces or synthetic data, scores heuristics-first then…

Add Opik tracing to an existing app and verify a real trace lands. Installs the Opik package, detects the language and LLM framework, adds the minimum tracing, runs a safe representative path, confirms a trace in Opik, and returns the trace link. Use for…

Improve a prompt with the Opik Agent Optimizer — resolve the prompt, a dataset, and a metric, pick the algorithm, run a bounded optimization, check the gain on held-out data, and save the winner as a new prompt version with the optimization run link. Runs via…

Decide ship or hold for a candidate from the compare skill's numbers, against an explicit release policy — regressions, pass rate, safety-tagged cases, subgroup consistency, latency and cost budgets, flakiness, evidence size, and whether the judge is…

Root-cause a specific Opik trace, or a pattern across traces, and return a grounded explanation. Uses the hosted Opik MCP when it is connected, and falls back to SDK scripting otherwise. Returns the root cause, the evidence spans as clickable Opik UI links,…

Take a judge live on production traffic — create an Opik online evaluation rule (LLM-as-judge or Python metric) on a project with sampling, filters, variable mapping, and a cost cap, then confirm new traces are being scored. Works over the SDK's REST client;…

Run a candidate against the baseline over an Opik test suite and read the numbers back — which cases broke, which got fixed, the per-metric deltas, worst rows, and whether the two runs are comparable — with the Opik compare-view link. Runs via the SDK; reads…

Build an LLM evaluation and run it against the app, returning an Opik experiment with scores and its link. Picks a test suite with judge assertions or a dataset with metrics, sources cases from traces or synthetic data, scores heuristics-first then…

Improve a prompt with the Opik Agent Optimizer — resolve the prompt, a dataset, and a metric, pick the algorithm, run a bounded optimization, check the gain on held-out data, and save the winner as a new prompt version with the optimization run link. Runs via…

Reference for the Opik SDK — tracing, span types, framework integrations, threads, and the prompt library (Python, TypeScript, REST). Use for "what span types exist", "how do I flush", "track_openai", "add OpikTracer", "version a prompt". To instrument a repo…

Decide ship or hold for a candidate from the compare skill's numbers, against an explicit release policy — regressions, pass rate, safety-tagged cases, subgroup consistency, latency and cost budgets, flakiness, evidence size, and whether the judge is…

Surface the Opik traces worth a developer's attention, ranked by signal — Diagnostics issues first, then errors, failed tool calls, latency, regressions, and low online-eval scores. With the Opik MCP connected it lists the project's agent_insights_issue…

Root-cause a specific Opik trace, or a pattern across traces, and return a grounded explanation. Uses the hosted Opik MCP when it is connected, and falls back to SDK scripting otherwise. Returns the root cause, the evidence spans as clickable Opik UI links,…

Add Opik tracing to an existing app and verify a real trace lands. Installs the Opik package, detects the language and LLM framework, adds the minimum tracing, runs a safe representative path, confirms a trace in Opik, and returns the trace link. Use for…

Pull 2+ Comet experiments via the Comet MCP server (or `comet_ml.API` as fallback) and render a side-by-side markdown table of metrics and hyperparameters. Highlights the winning value per metric row. Use when the user asks to "compare", "diff", "which run…

Query a single Comet experiment via the Comet MCP server (or `comet_ml.API` as a fallback), fetch its metrics and parameters, and compare against thresholds (inline or from `thresholds.yaml`). Reports pass/fail per metric with a markdown table. Use when the…

Manage a Comet Model Registry version's status and/or tags via `scripts/registry_mutate.py`. Status is a single-value enum (None / Development / Staging / QA / Production by default; workspace-configurable). Tags are a multi-value free-form list. Confirms…

Execute `make train`, capture output, and surface the Comet experiment URL, registered models, logged artifacts, and final metrics. Use when the user asks to "run a training run", "train the model", or "kick off an experiment". Pure shell wrapper — does not…

Diff HEAD against main, group the changes by Keep-a-Changelog type (Added/Changed/Fixed/...), propose one-line entries under [Unreleased] in CHANGELOG.md, then edit the file inline. User reviews the diff and stages it themselves. Use before any commit that…

Audit the Makefile against the tooling actually installed (ruff, pytest, mypy, etc.) and the scripts under src/<module>/, then propose missing targets or fix stale ones. Use when adding a new tool or script that should be invokable via make. Preserves…

Regenerate the auto-generated sections of README.md (project tree, common-tasks table) to reflect the current state of pyproject.toml, the Makefile, and src/. Use when project structure changes, deps change, or those sections drift from reality. Preserves any…

Scaffold a brand-new Opik example in this repo from a canonical template under templates/ — either a full use-case demo (templates/use-case-template/) or a standalone utility script (templates/script-template/). Use this whenever the user wants to create,…

Add new items to an Opik evaluation dataset in this repo. Use when the user says "add a case to the dataset", "add items to the Opik dataset", "extend the eval dataset", "seed the dataset", "add test data for the eval", or names new query/expected-output…

Add new items (plain-English assertions) to an Opik test suite in this repo. Use when the user says "add an assertion", "add a test case to the suite", "extend the eval suite", "add a check to the test suite", "the model should also be tested for X", or…

Run an Opik evaluation for an example in this repo — metrics (evaluate) and/or assertions (run_tests) against a dataset/test suite. Use when the user says "run the eval", "evaluate the example", "run the test suite", "score the model", "check the metrics",…

Run an Opik prompt optimization (opik-optimizer) for an example in this repo, and optionally promote the result to the Prompt Library. Use when the user says "optimize the prompt", "run the optimizer", "improve the prompt", "run Optimization Studio", "tune…

Surface the Opik traces worth a developer's attention, ranked by signal — Diagnostics issues first, then errors, failed tool calls, latency, regressions, and low online-eval scores. With the Opik MCP connected it lists the project's agent_insights_issue…

Root-cause a specific Opik trace, or a pattern across traces, and return a grounded explanation. Uses the hosted Opik MCP when it is connected, and falls back to SDK scripting otherwise. Returns the root cause, the evidence spans as clickable Opik UI links,…

Add Opik tracing to an existing app and verify a real trace lands. Installs the Opik package, detects the language and LLM framework, adds the minimum tracing, runs a safe representative path, confirms a trace in Opik, and returns the trace link. Use for…

Scaffold a brand-new Comet example in this repo from the canonical template under templates/integration-example/. Use whenever the user wants to create, add, or start a new example, demo, or integration on Comet — phrasings like "add a Comet example for X",…

How the opik-cipx cost-tracking proxy works — architecture, the CLI, where it stores state, how it's enabled and disabled, privacy and telemetry, and how to read `opik-cipx status`. Use whenever the user asks about opik-cipx, token/cost tracking, or Opik…