| name | mkt-experiment |
| description | Autonomous growth experimentation framework. Creates A/B/multivariate experiments with hypotheses, logs data points, runs statistical analysis (bootstrap CI + Mann-Whitney U), auto-promotes winners to a living playbook, and suggests next experiments. Use when creating or managing marketing experiments, logging data points, scoring experiments, or generating weekly scorecards. |
Growth Engine
Autonomous growth experimentation framework based on Karpathy's autoresearch pattern applied to marketing. Creates experiments with hypotheses, logs data points, runs statistical analysis (bootstrap CI + Mann-Whitney U), auto-promotes winners to a living playbook, and suggests next experiments. Supports batch mode (up to 10 variants simultaneously).
Usage
Use this skill when:
- Creating or managing A/B or multivariate experiments for any marketing channel
- Logging experiment data points after content is published or campaigns run
- Scoring experiments to determine statistical winners
- Checking the playbook for proven best practices before creating new content
- Generating weekly scorecards across all channels
- Monitoring campaign pacing and health
Do NOT use for:
- One-off content creation (use the playbook output as input, but don't run the engine)
- Non-experiment analytics or reporting
- Campaign setup in external platforms (this tracks experiments, not campaign config)
Commands
Create an experiment
python3 experiment-engine.py create \
--agent <agent_name> \
--hypothesis "What you expect to happen" \
--variable "<variable_name>" \
--variants '["variant_a", "variant_b"]' \
--metric "<primary_metric>" \
--cycle-hours 24
Add --batch-mode for 3-10 variant tests. Add --min-samples N to override auto-detection.
Log a data point
python3 experiment-engine.py log \
--agent <agent_name> \
--experiment-id <EXP-ID> \
--variant "<variant_name>" \
--metrics '{"metric_name": value}'
Score an experiment
python3 experiment-engine.py score --agent <agent_name> --experiment-id <EXP-ID>
Statuses: running → trending → keep (winner) or discard (loser)
Winners auto-promote to the playbook. Requires p < 0.05 AND ≥ 15% lift.
List experiments
python3 experiment-engine.py list --agent <agent_name> [--status running|trending|keep|discard]
Check the playbook
python3 experiment-engine.py playbook --agent <agent_name>
Always check the playbook before creating new content to apply proven best practices.
Suggest next experiments
python3 experiment-engine.py suggest --agent <agent_name>
Generate weekly scorecard
python3 autogrowth-weekly-scorecard.py [--weeks N] [--output file.md]
Check campaign pacing
python3 pacing-alert.py [--json]
Exit code 0 = on pace, 1 = alerts present.
Workflow
- Before creating content:
playbook → apply proven rules
- When publishing:
log → record which variant was used and its metrics
- Periodically:
score → check if experiments have reached statistical significance
- Weekly:
autogrowth-weekly-scorecard.py → review all channels
- After completing experiments:
suggest → pick the next variable to test
Configuration
Required Environment Variables
| Variable | Description |
|---|
GROWTH_ENGINE_DATA_DIR | Data directory (default: ./data/experiments) |
GROWTH_ENGINE_AGENTS | Comma-separated agent names (default: content,email,linkedin,seo,blog) |
Optional Tuning
| Variable | Default | Description |
|---|
HIGH_VOLUME_AGENTS | content,email | Agents needing only 10 samples/variant |
LOW_VOLUME_AGENTS | seo,linkedin,blog | Agents needing 30 samples/variant |
P_WINNER | 0.05 | p-value threshold for winner |
P_TREND | 0.10 | p-value threshold for trending |
LIFT_WIN | 15.0 | Minimum % lift for keep decision |
BOOTSTRAP_ITERATIONS | 1000 | Bootstrap resamples for CI |
BATCH_MODE_MAX_VARIANTS | 10 | Max variants in batch mode |
Pacing Alert Variables
| Variable | Description |
|---|
PIPELINE_API_URL | Pipeline/CRM API endpoint |
PIPELINE_AUTH_TOKEN | Bearer token for pipeline API |
RECRUITING_API_URL | Recruiting API endpoint |
RECRUITING_AUTH_TOKEN | Bearer token for recruiting API |
EMAIL_API_URL | Email platform API base URL |
EMAIL_AUTH_TOKEN | Bearer token for email platform |
OUTBOUND_CAMPAIGNS | JSON: {"name": "campaign-id"} |
RECRUITING_CAMPAIGNS | JSON: {"name": "campaign-id"} |
DAILY_LEAD_TARGET | Leads/day target (default: 10) |
WEEKLY_CANDIDATE_TARGET | Candidates/week target (default: 400) |
Dependencies
pip install numpy scipy
