メニュー
SkillsMP は braintrustdata/eval-library から 24 件の skill を収集しています。skill を開くとソースと詳細を確認できます。
収集済み skill 24 件中 24 件を表示しています。
Analyze completed LLM or agent eval experiments using uncertainty-aware and decision-relevant methods. Use to audit run completeness and pairing, calculate confidence intervals, run paired comparisons, report wins, losses, and ties, incorporate run-to-run…
原文の言語: 英語
Attribute an observed change when several things moved at once — model plus prompt plus tools, a provider migration, a framework upgrade, or a vendor swap that bundles serving stack with model. Use when asked which part of a change caused the result, when a…
原文の言語: 英語
Create, edit, audit, or compare eval datasets for LLM applications and agents, including target-population definition, case sourcing from production traces, stratified sampling, label provenance and label audits, expected values as constraints for open-ended…
原文の言語: 英語
Create, edit, or audit an eval objective by working backward from a product decision to the target outcome, construct, population, intended claim, and verification-versus-validation questions. Use when a team is unsure what an eval should establish, asks…
原文の言語: 英語
Create, edit, audit, or apply release gates for LLM applications and agents. Use to combine minimum meaningful improvement, statistical significance, regression rate, subgroup consistency, worst-run stability, all-attempts reliability, safety upper bounds,…
原文の言語: 英語
Take a validated scorer or classifier from definition to running instrument in Braintrust — scope selection, inline testing before saving, saving as an evaluator, attaching an online-scoring rule, activating it for new traffic, and backfilling history with a…
原文の言語: 英語
Design or audit controlled eval experiments for model, prompt, retrieval, tool, guardrail, or agent-architecture changes. Use before data collection to state directional and minimum-effect hypotheses, name independent, dependent, and control variables…
原文の言語: 英語
Design the trace and eval-dataset schema for an LLM app or agent, and wire the system to emit it. Use when deciding what to log, designing a trace schema, setting up tracing or observability before evals, or when failures cannot be debugged or sliced from…
原文の言語: 英語
Create, edit, audit, or compare a multi-objective eval metric bundle covering product quality, safety, reliability, latency, and cost. Use to choose metrics for an eval, define a goodness bundle, distinguish optimization metrics from non-regression…
原文の言語: 英語
Design or audit human evaluation workflows and golden datasets for LLM applications and agents. Use to set up expert review, select review cases, write reviewer instructions, assign raters, capture rationales and confidence, measure inter-rater agreement with…
原文の言語: 英語
Search open-endedly for unanticipated agent failure modes and convert them into a named taxonomy and durable regression items. Use for requests to find out what goes wrong, surface unknown or silent failures, do error analysis over traces, cluster and triage…
原文の言語: 英語
Design, evaluate, and deploy a Topics pipeline that discovers clusters in trace traffic — preprocessor, facet prompt, no-match policy, clustering, and the automation that runs it. Use when the label set is unknown and has to come out of the data: finding what…
原文の言語: 英語
Extract evaluation criteria out of domain experts and real user desires, and capture them as reusable evaluation assets before any labeling or scoring begins — construct facets, anchored exemplars, adversarial traps, scoring guidance, audit rules, and the…
原文の言語: 英語
Create or audit an evidence map that connects eval constructs or behavior specifications to observable success signals, failure signals, proxy limitations, and possible gaming paths. Use when a quality such as helpfulness, safety, trust, correctness, or task…
原文の言語: 英語
Design or audit online evaluation of live LLM and agent traffic: trace sampling, online scoring coverage, alert thresholds and ownership, drift and failure-slice monitoring, incident review, and the pipeline that routes production failures back into the…
原文の言語: 英語
Plan an evaluation workflow for an LLM application or agent by identifying the product decision, expected behavior, target population, evidence, datasets, scorers, validation, analysis, and release criteria. Use for broad or early-stage requests such as "help…
原文の言語: 英語
Run the same dataset under controlled variants to measure either the ceiling of what a system can do or the spread of how reliably it does it. Use for questions about sensitivity to prompt paraphrases, formatting, ordering, seeds, or repeated runs; output…
原文の言語: 英語
Plan and run adversarial worst-case testing of an authorized LLM or agent system against an explicit threat model, policy boundary, or high-impact prohibited behavior. Use for requests to red team, attack, jailbreak-test, or stress a system's guardrails, to…
原文の言語: 英語
Turn completed eval analysis into a technical report, release summary, chart set, or decision document with claims calibrated to the evidence. Use when writing up eval or benchmark results, preparing charts or tables of model comparisons, drafting a release…
原文の言語: 英語
Calculate or audit eval sample sizes, minimum detectable effects, confidence interval precision, required repeated runs, and clean-trial counts for bounding rare failures. Use when a user asks how many eval cases, items, scenarios, runs, or safety trials are…
原文の言語: 英語
Validate automated eval scorers and LLM judges against expert-reviewed reference data. Use to compare scorer output with human labels, calculate agreement (kappa, alpha) with uncertainty, inspect confusion by class and severity, analyze subgroup failures,…
原文の言語: 英語
Create, edit, audit, or compare versioned Agent Behavior specifications in .agents/behaviors/<name>/BEHAVIOR.md. Use when a user wants to define recurring agent conduct, write a behavioral contract, make implicit trace-review expectations explicit, specify…
原文の言語: 英語
Design, implement, edit, or audit narrow eval scorers for LLM applications and agents, including deterministic checks, reference or final-state comparisons, trace and tool-call checks, and anchored LLM-as-judge rubrics. Use when translating one observable…
原文の言語: 英語
Execute an eval end to end against a live Braintrust project — org and credential selection, finding and importing a dataset, writing the task and scorer code, running the experiment behind smoke gates and quota preflight, and tracing agentic `claude -p`…
原文の言語: 英語