comet-ml

Local development environment setup and commands. Use when helping with dev server, Docker, or local testing.

Build, update, test, and document Opik SDK integrations (Python & TypeScript). Use when adding a new framework/provider integration under sdks/python/src/opik/integrations or sdks/typescript/src/opik/integrations, updating an existing one, or verifying that an integration logs traces correctly.

Recipe for wiring a new linter into Opik's unified 🐙 Code Quality pipeline (pre-commit + CI). Use when adding a pre-commit-driven linter/formatter to the repo — enumerates every file that must change (`.pre-commit-config.yaml`, `scripts/precommit-hook-descriptions.tsv`, `scripts/precommit-detect-hooks.py`, `CONTRIBUTING.md`), the non-obvious gotchas (mandatory explicit `files:`, the blank-description trap, `TOOLCHAIN_BY_ID`/`TYPED_IDS`), the retroactive fix-vs-suppress policy, and the pass/fail verification loop. Worked examples: actionlint (live) and hadolint (OPIK-6673).

Use when building or extending a Page Object Model (POM) for the Opik E2E suite (under `tests_end_to_end/e2e/pom/`) and you need to choose stable selectors against the live UI. Walks through seeding required state, exploring the running page with the Playwright MCP (accessibility snapshot + data-testid enumeration), picking the most stable locator for each element, and verifying it before committing. Used as the discovery sub-step by the `writing-e2e-tests` skill.

Specification for instrumenting an opik-backend workflow with operational OpenTelemetry metrics — per-stage throughput/latency/error counters and native histograms, dimensioned per-customer (workspace). Use when a pipeline (scoring, ingestion, experiments, jobs) needs per-stage visibility. Covers metric emission only; building the Grafana dashboard from these metrics is a separate skill. Distinct from analytics-instrumentation (PostHog product events).

Build or update an Opik integration that lives OUTSIDE this repo — a standalone opik-* package (e.g. opik-openclaw, opik-claude-code-plugin) or Opik support contributed into a third-party project (e.g. LiteLLM, Dify). Use ONLY when the user names an external repository or external package as the target; for integrations under sdks/ use the opik-integrations skill instead.

Use when a developer wants to add, write, or create an end-to-end test for an Opik feature, page, or branch — e.g. "add an e2e test for the experiments comparison page", "write a test for the feature I just built", "e2e test for this branch", "cover the dataset items flow with a test". Runs the full loop in tests_end_to_end/e2e/ — analyze the feature and frontend code, explore the live UI with the Playwright MCP, write the Page Object Model + spec, and run it locally until green.

Use when an Opik E2E test has failed and a developer wants it investigated — e.g. "why did this e2e test fail?", "investigate the failing run on my PR", "is dataset-crud-smoke flaky?", "the nightly e2e suite went red". Takes a failure from a CI check, a TestOps launch, a test name, or a local run; gathers the trace and history, classifies regression vs. flake, and proposes a fix. Read-only — it diagnoses and proposes, it does not edit tests.

Pull 2+ Comet experiments via the Comet MCP server (or `comet_ml.API` as fallback) and render a side-by-side markdown table of metrics and hyperparameters. Highlights the winning value per metric row. Use when the user asks to "compare", "diff", "which run won", or "baseline vs new".

Query a single Comet experiment via the Comet MCP server (or `comet_ml.API` as a fallback), fetch its metrics and parameters, and compare against thresholds (inline or from `thresholds.yaml`). Reports pass/fail per metric with a markdown table. Use when the user asks to "evaluate", "gate", or "check" a specific run before promotion.

Manage a Comet Model Registry version's status and/or tags via `scripts/registry_mutate.py`. Status is a single-value enum (None / Development / Staging / QA / Production by default; workspace-configurable). Tags are a multi-value free-form list. Confirms before mutating. Optionally gates on a passing `evaluate-run` outcome. Use when the user asks to "promote", "demote", "set status", "tag", "untag", or "retag" a model.

Execute `make train`, capture output, and surface the Comet experiment URL, registered models, logged artifacts, and final metrics. Use when the user asks to "run a training run", "train the model", or "kick off an experiment". Pure shell wrapper — does not use the Comet MCP server.

Diff HEAD against main, group the changes by Keep-a-Changelog type (Added/Changed/Fixed/...), propose one-line entries under [Unreleased] in CHANGELOG.md, then edit the file inline. User reviews the diff and stages it themselves. Use before any commit that touches src/.

Audit the Makefile against the tooling actually installed (ruff, pytest, mypy, etc.) and the scripts under src/<module>/, then propose missing targets or fix stale ones. Use when adding a new tool or script that should be invokable via make. Preserves user-added custom targets.

Regenerate the auto-generated sections of README.md (project tree, common-tasks table) to reflect the current state of pyproject.toml, the Makefile, and src/. Use when project structure changes, deps change, or those sections drift from reality. Preserves any user-authored prose outside the AUTO-GENERATED markers.

Scaffold a brand-new Opik example in this repo from a canonical template under templates/ — either a full use-case demo (templates/use-case-template/) or a standalone utility script (templates/script-template/). Use this whenever the user wants to create, add, or start a new use-case, demo, example, example project, or Opik utility script — phrasings like "create a new use case", "add a demo for X", "scaffold an Opik example", "start a new demo like f1_radio_rag", "set up a new example project", "add a script that does X", or describes a new application or utility they want to build on Opik. Do NOT hand-build the folder structure or copy files manually — this skill stamps out the project and renames it correctly. Require a name and a one-line description of what it does; refuse to invent a demo from nothing.

Add new items to an Opik evaluation dataset in this repo. Use when the user says "add a case to the dataset", "add items to the Opik dataset", "extend the eval dataset", "seed the dataset", "add test data for the eval", or names new query/expected-output pairs to evaluate. Appends to the example's canonical source data file first, then syncs to Opik via get_or_create_dataset + dataset.insert. Require a target example (or dataset name) and the item content; refuse to invent dataset rows.

Add new items (plain-English assertions) to an Opik test suite in this repo. Use when the user says "add an assertion", "add a test case to the suite", "extend the eval suite", "add a check to the test suite", "the model should also be tested for X", or describes a pass/fail criterion for an example's output. Appends to the canonical source file, then syncs via get_or_create_test_suite + suite.insert. Require a target example/suite and the assertion text; refuse to invent checks.

Run an Opik evaluation for an example in this repo — metrics (evaluate) and/or assertions (run_tests) against a dataset/test suite. Use when the user says "run the eval", "evaluate the example", "run the test suite", "score the model", "check the metrics", "what's the pass rate", or wants to measure an example's quality in Opik. Covers the credentials/DRY_RUN gate, metric selection, experiment naming, and reading results. Require a target example; refuse to guess which one.

Run an Opik prompt optimization (opik-optimizer) for an example in this repo, and optionally promote the result to the Prompt Library. Use when the user says "optimize the prompt", "run the optimizer", "improve the prompt", "run Optimization Studio", "tune the prompt", or wants a measured better prompt for an example. Covers the metric-callable shape, reading the score delta + run link, and versioning the winner. Require a target example with an eval dataset; refuse to optimize without a dataset to score against.

This skill should be used when the user needs to add Opik tracing or integrations to their code, instrument an LLM application, or needs reference for Opik SDK usage (Python, TypeScript, REST API). Use for tasks like "add tracing", "instrument my code", "use track_openai", "add OpikTracer", "what span types are available", "how to flush traces".

This skill should be used when the user asks about agent architecture, evaluation, metrics, production monitoring, debugging agents, or best practices for building reliable AI agents. Use for questions like "evaluate my agent", "set up production monitoring", "add guardrails", "detect hallucinations", "agent anti-patterns", "compare experiments", "create evaluation dataset".

Add Opik tracing to an existing codebase. Detects language (Python/TypeScript), identifies LLM frameworks, adds appropriate decorators and integrations, marks entrypoints, and wires up environment config. Use for "instrument my code", "add opik tracing", "add observability", or "trace my agent".

Opik observability for LLM agents — Prompt Library, Local Runner (opik connect), Test Suites, threads, integrations. Use for "manage my prompts", "connect my agent", "evaluate my agent" or "integrate with Opik".

Scaffold a brand-new Comet example in this repo from the canonical template under templates/integration-example/. Use whenever the user wants to create, add, or start a new example, demo, or integration on Comet — phrasings like "add a Comet example for X", "scaffold an integration example", "create a new example for <framework>", "start a hello-world for <library>", "set up a new example project". Do NOT hand-build the folder structure or copy files manually — this skill stamps out the example and renames it correctly. Require a name and a one-line description of what it does; refuse to invent an example from nothing.

How the opik-cipx cost-tracking proxy works — architecture, the CLI, where it stores state, how it's enabled and disabled, privacy and telemetry, and how to read `opik-cipx status`. Use whenever the user asks about opik-cipx, token/cost tracking, or Opik spans, or when spans stop reaching Opik, Claude Code seems to bypass the proxy, or you need to explain, diagnose, or repair any part of it.