메뉴
Experiment orchestration framework with traffic allocation, statistical analysis, and early stopping detection. Use to test routing changes, model upgrades, and role assignments with Welch's t-test significance testing.
Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.
SOC 직업 분류 기준
DEPRECATED — This skill is no longer maintained. Prometheus and Grafana infrastructure was never implemented. Use local JSON metrics analysis instead.
Scaffolds new SPEC-compliant agentic-engineers agents with a single call. Generates SKILL.md frontmatter, test scaffolds (TDD RED-phase), __init__.py, scripts/ layout, and DELEGATE/HANDBACK protocol templates. Use when creating any new automation agent, task handler, or operational tool. Validates role, model, effort, naming, and dependency graphs (circular dep detection) before writing files. Supports dry-run mode for planning without side effects.
Routine maintenance for Codex sessions: monitor queue state, close completed sub-agents, resume or escalate active work, and keep agent capacity available.
Automated cross-validation of protocol queue integrity. Scans all DELEGATEs/HANDBACKs, validates schema compliance, detects cycles, checks rate limits, generates compliance report. Enables self-referential protocol improvements.
Consolidates provider-specific AI costs into unified metrics across Anthropic, OpenAI, Google Gemini, GitHub Copilot, and Ollama. Enables apples-to-apples cost comparison and savings analysis.
Cost Budget Enforcement (COST-001) — enforces per-session, per-hour, and per-day AI spend limits across all agents and models. Blocks task execution when budgets are exceeded and emits structured alerts for operator action. Integrates with model-selection (COST-003) to pre-screen model choices against budget headroom.
| name | ab-testing |
| description | Experiment orchestration framework with traffic allocation, statistical analysis, and early stopping detection. Use to test routing changes, model upgrades, and role assignments with Welch's t-test significance testing. |
| license | Proprietary |
| compatibility | Designed for agentic-engineers framework |
| metadata | {"author":"agentic-engineers","version":"1.0","category":"optimization","role":"lead-engineer","schedule":"0 18 * * *"} |
A/B Testing Monitor conducts controlled experiments on task routing, model allocation, and role assignments with statistical rigor.
What it does:
python scripts/ab-testing.py --create --name "Test name" --hypothesis "..."
python scripts/ab-testing.py --start EXPERIMENT_ID
python scripts/ab-testing.py --analyze EXPERIMENT_ID
python scripts/ab-testing.py --stop EXPERIMENT_ID
Daily at 18:00 UTC (for early stopping checks) via orchestration/config/ab-testing-monitor.cron
0 18 * * * cd <project-root> && python agentic-engineers/skills/ab-testing/scripts/ab-testing.py --monitor
agentic-engineers/data/experiments/Input: Model Engineer proposals + metrics
Output: Experiment results and winner determination
Dashboard: A/B Testing (Grafana)
Winner implementation requires manual approval and deployment.
ab-testing.py — Main orchestration engineThis skill participates in the framework's continuous improvement cycle (see skill-improvement-feedback).
When you use ab-testing during a task, include a skill_feedback entry in your HANDBACK to help improve it over time:
skill_feedback:
- skill_name: ab-testing
effectiveness_score: 0.85 # required: 0.0–1.0
clarity_score: 0.90 # optional
coverage_gaps:
- "Specific scenario the skill did not address"
improvement_suggestions:
- "Concrete change that would have helped"
usage_context: "One sentence on how you used this skill"
Positive feedback is as valuable as critical feedback. Three or more feedback items for this skill automatically trigger an improvement task.