メニュー
Experiment orchestration framework with traffic allocation, statistical analysis, and early stopping detection. Use to test routing changes, model upgrades, and role assignments with Welch's t-test significance testing.
Codex または Claude でインストール この Prompt をコピーして Codex、Claude、または他のアシスタントに貼り付けると、Skill ページを確認してインストールできます。
SOC 職業分類に基づく
DEPRECATED — This skill is no longer maintained. Prometheus and Grafana infrastructure was never implemented. Use local JSON metrics analysis instead.
Scaffolds new SPEC-compliant agentic-engineers agents with a single call. Generates SKILL.md frontmatter, test scaffolds (TDD RED-phase), __init__.py, scripts/ layout, and DELEGATE/HANDBACK protocol templates. Use when creating any new automation agent, task handler, or operational tool. Validates role, model, effort, naming, and dependency graphs (circular dep detection) before writing files. Supports dry-run mode for planning without side effects.
Routine maintenance for Codex sessions: monitor queue state, close completed sub-agents, resume or escalate active work, and keep agent capacity available.
Automated cross-validation of protocol queue integrity. Scans all DELEGATEs/HANDBACKs, validates schema compliance, detects cycles, checks rate limits, generates compliance report. Enables self-referential protocol improvements.
Consolidates provider-specific AI costs into unified metrics across Anthropic, OpenAI, Google Gemini, GitHub Copilot, and Ollama. Enables apples-to-apples cost comparison and savings analysis.
Cost Budget Enforcement (COST-001) — enforces per-session, per-hour, and per-day AI spend limits across all agents and models. Blocks task execution when budgets are exceeded and emits structured alerts for operator action. Integrates with model-selection (COST-003) to pre-screen model choices against budget headroom.
| name | ab-testing |
| description | Experiment orchestration framework with traffic allocation, statistical analysis, and early stopping detection. Use to test routing changes, model upgrades, and role assignments with Welch's t-test significance testing. |
| license | Proprietary |
| compatibility | Designed for agentic-engineers framework |
| metadata | {"author":"agentic-engineers","version":"1.0","category":"optimization","role":"lead-engineer","schedule":"0 18 * * *"} |
A/B Testing Monitor conducts controlled experiments on task routing, model allocation, and role assignments with statistical rigor.
What it does:
python scripts/ab-testing.py --create --name "Test name" --hypothesis "..."
python scripts/ab-testing.py --start EXPERIMENT_ID
python scripts/ab-testing.py --analyze EXPERIMENT_ID
python scripts/ab-testing.py --stop EXPERIMENT_ID
Daily at 18:00 UTC (for early stopping checks) via orchestration/config/ab-testing-monitor.cron
0 18 * * * cd <project-root> && python agentic-engineers/skills/ab-testing/scripts/ab-testing.py --monitor
agentic-engineers/data/experiments/Input: Model Engineer proposals + metrics
Output: Experiment results and winner determination
Dashboard: A/B Testing (Grafana)
Winner implementation requires manual approval and deployment.
ab-testing.py — Main orchestration engineThis skill participates in the framework's continuous improvement cycle (see skill-improvement-feedback).
When you use ab-testing during a task, include a skill_feedback entry in your HANDBACK to help improve it over time:
skill_feedback:
- skill_name: ab-testing
effectiveness_score: 0.85 # required: 0.0–1.0
clarity_score: 0.90 # optional
coverage_gaps:
- "Specific scenario the skill did not address"
improvement_suggestions:
- "Concrete change that would have helped"
usage_context: "One sentence on how you used this skill"
Positive feedback is as valuable as critical feedback. Three or more feedback items for this skill automatically trigger an improvement task.