| name | ab-testing |
| description | Experiment orchestration framework with traffic allocation, statistical analysis, and early stopping detection. Use to test routing changes, model upgrades, and role assignments with Welch's t-test significance testing. |
| license | Proprietary |
| compatibility | Designed for agentic-engineers framework |
| metadata | {"author":"agentic-engineers","version":"1.0","category":"optimization","role":"lead-engineer","schedule":"0 18 * * *"} |
Overview
A/B Testing Monitor conducts controlled experiments on task routing, model allocation, and role assignments with statistical rigor.
What it does:
- Creates experiments from proposals
- Allocates traffic (50/50 control/variant)
- Monitors results (daily statistical checks)
- Analyzes significance (Welch's t-test, p < 0.05)
- Determines winners (cost + quality)
- Early stops (if clear winner or regression)
Invocation
Manual Run
python scripts/ab-testing.py --create --name "Test name" --hypothesis "..."
python scripts/ab-testing.py --start EXPERIMENT_ID
python scripts/ab-testing.py --analyze EXPERIMENT_ID
python scripts/ab-testing.py --stop EXPERIMENT_ID
Automated (Cron)
Daily at 18:00 UTC (for early stopping checks) via orchestration/config/ab-testing-monitor.cron
0 18 * * * cd <project-root> && python agentic-engineers/skills/ab-testing/scripts/ab-testing.py --monitor
Configuration
Configuration
- Experiments stored:
agentic-engineers/data/experiments/
- Min sample size: 100 tasks per group
- Significance threshold: p < 0.05 (95% confidence)
- Default duration: 7 days
- Traffic split: 50/50 control/variant
Integration
Input: Model Engineer proposals + metrics
Output: Experiment results and winner determination
Dashboard: A/B Testing (Grafana)
Winner implementation requires manual approval and deployment.
Scripts
ab-testing.py — Main orchestration engine
Self-Improvement
This skill participates in the framework's continuous improvement cycle
(see skill-improvement-feedback).
When you use ab-testing during a task, include a skill_feedback entry
in your HANDBACK to help improve it over time:
skill_feedback:
- skill_name: ab-testing
effectiveness_score: 0.85
clarity_score: 0.90
coverage_gaps:
- "Specific scenario the skill did not address"
improvement_suggestions:
- "Concrete change that would have helped"
usage_context: "One sentence on how you used this skill"
Positive feedback is as valuable as critical feedback. Three or more
feedback items for this skill automatically trigger an improvement task.
