Align, calibrate, or improve an existing binary Pass/Fail LLM-as-a-judge (orq evaluator) so its verdicts match human judgment. Use when the user wants to "align my evaluator", "improve my eval", "annotate an evaluator", "find ambiguous cases", or "build an annotation queue" — i.e. they have a boolean judge (evaluator = LLM-as-a-judge) that disagrees with human labels or is inconsistent. Measures judge self-consistency (flip-rate) via repeated runs, surfaces the most ambiguous datapoints for human annotation, rewrites the judge prompt from the labels, and creates the new evaluator only after the human approves. If the evaluator ID isn't given, ask for it after triggering. Do NOT use to build an evaluator from scratch (use orq-build-evaluator), to fix failures with prompt tweaks (use orq-optimize-prompt), or for non-boolean judges.
Manage orq.ai Skills (the platform entity, formerly called Snippets) end-to-end — list, get, create, update, and delete Skills, plus authoring guidance (display name, description, tags, project scoping, path placement), and how Skills get consumed (the `{{skill.<display_name>}}` template placeholder inside prompts and agent instructions). Use when the user wants to create, audit, edit, retire, or hook up orq.ai Skills.
Write and run evaluatorq evaluation scripts (Python or TypeScript) for a single agent or deployment — custom scorers, built-in evaluators, and dataset-driven evaluation. For CLI workflows, use the companion skills: `orq-red-team` for `eq redteam` adversarial testing and `orq-simulate-agent` for `eq sim` multi-turn user simulation. Do NOT use when comparing multiple agents head-to-head (use orq-compare-agents) or when running orq.ai-native experiments only (use orq-run-experiment).
Run cross-framework agent comparisons using evaluatorq from orqkit — compares any combination of agents (orq.ai, LangGraph, CrewAI, OpenAI Agents SDK, Vercel AI SDK) head-to-head on the same dataset with LLM-as-a-judge scoring. Use when comparing agents, benchmarking, or wanting side-by-side evaluation. Do NOT use when comparing only orq.ai configurations with no external agents (use orq-run-experiment instead).
Invoke the evaluatorq red teaming CLI to run adversarial attacks against deployed agents or static datasets. Use when asked to "run a red team", "red team this deployment", "test my agent for vulnerabilities", "OWASP red team", or "check how my agent handles adversarial inputs". Do NOT use when you only need to build evaluators (use orq-build-evaluator) or analyze existing trace failures (use orq-analyze-trace-failures).
Read production traces, identify what's failing, and build failure taxonomies using open coding and axial coding methodology. Use when debugging agent or pipeline quality, investigating "why are my outputs bad?", or before building any evaluator — error analysis must come first. Do NOT use when you already have identified failure modes and need evaluators (use orq-build-evaluator) or datasets (use orq-generate-synthetic-dataset).
Design, create, and configure orq.ai Agents with tools, instructions, knowledge bases, and memory stores. Use when building new agents, attaching KBs or memory, writing system instructions, selecting models, or setting up RAG pipelines. Do NOT use for debugging existing agents (use orq-analyze-trace-failures) or comparing agents across frameworks (use orq-compare-agents).
Create validated LLM-as-a-Judge evaluators following best practices — binary Pass/Fail judges with TPR/TNR validation for measuring specific failure modes. Use when you need to automate quality checks, build guardrails, or measure a specific failure mode identified during trace analysis. Do NOT use when failures are fixable with prompt changes (use orq-optimize-prompt) or when failure modes are unknown (use orq-analyze-trace-failures first).