mkt-experiment

name	mkt-experiment
description	Autonomous growth experimentation framework. Creates A/B/multivariate experiments with hypotheses, logs data points, runs statistical analysis (bootstrap CI + Mann-Whitney U), auto-promotes winners to a living playbook, and suggests next experiments. Use when creating or managing marketing experiments, logging data points, scoring experiments, or generating weekly scorecards.

Growth Engine

Autonomous growth experimentation framework based on Karpathy's autoresearch pattern applied to marketing. Creates experiments with hypotheses, logs data points, runs statistical analysis (bootstrap CI + Mann-Whitney U), auto-promotes winners to a living playbook, and suggests next experiments. Supports batch mode (up to 10 variants simultaneously).

Usage

Use this skill when:

• Creating or managing A/B or multivariate experiments for any marketing channel
• Logging experiment data points after content is published or campaigns run
• Scoring experiments to determine statistical winners
• Checking the playbook for proven best practices before creating new content
• Generating weekly scorecards across all channels
• Monitoring campaign pacing and health

Do NOT use for:

• One-off content creation (use the playbook output as input, but don't run the engine)
• Non-experiment analytics or reporting
• Campaign setup in external platforms (this tracks experiments, not campaign config)

Commands

Create an experiment

python3 experiment-engine.py create \
  --agent <agent_name> \
  --hypothesis "What you expect to happen" \
  --variable "<variable_name>" \
  --variants '["variant_a", "variant_b"]' \
  --metric "<primary_metric>" \
  --cycle-hours 24

Add --batch-mode for 3-10 variant tests. Add --min-samples N to override auto-detection.

Log a data point

python3 experiment-engine.py log \
  --agent <agent_name> \
  --experiment-id <EXP-ID> \
  --variant "<variant_name>" \
  --metrics '{"metric_name": value}'

Score an experiment

python3 experiment-engine.py score --agent <agent_name> --experiment-id <EXP-ID>

Statuses: running → trending → keep (winner) or discard (loser)

Winners auto-promote to the playbook. Requires p < 0.05 AND ≥ 15% lift.

List experiments

python3 experiment-engine.py list --agent <agent_name> [--status running|trending|keep|discard]

Check the playbook

python3 experiment-engine.py playbook --agent <agent_name>

Always check the playbook before creating new content to apply proven best practices.

Suggest next experiments

python3 experiment-engine.py suggest --agent <agent_name>

Generate weekly scorecard

python3 autogrowth-weekly-scorecard.py [--weeks N] [--output file.md]

Check campaign pacing

python3 pacing-alert.py [--json]

Exit code 0 = on pace, 1 = alerts present.

Workflow

1. Before creating content: playbook → apply proven rules
2. When publishing: log → record which variant was used and its metrics
3. Periodically: score → check if experiments have reached statistical significance
4. Weekly: autogrowth-weekly-scorecard.py → review all channels
5. After completing experiments: suggest → pick the next variable to test

Configuration

Required Environment Variables

Variable	Description
GROWTH_ENGINE_DATA_DIR	Data directory (default: ./data/experiments)
GROWTH_ENGINE_AGENTS	Comma-separated agent names (default: content,email,linkedin,seo,blog)

Optional Tuning

Variable	Default	Description
HIGH_VOLUME_AGENTS	content,email	Agents needing only 10 samples/variant
LOW_VOLUME_AGENTS	seo,linkedin,blog	Agents needing 30 samples/variant
P_WINNER	0.05	p-value threshold for winner
P_TREND	0.10	p-value threshold for trending
LIFT_WIN	15.0	Minimum % lift for keep decision
BOOTSTRAP_ITERATIONS	1000	Bootstrap resamples for CI
BATCH_MODE_MAX_VARIANTS	10	Max variants in batch mode

Pacing Alert Variables

Variable	Description
PIPELINE_API_URL	Pipeline/CRM API endpoint
PIPELINE_AUTH_TOKEN	Bearer token for pipeline API
RECRUITING_API_URL	Recruiting API endpoint
RECRUITING_AUTH_TOKEN	Bearer token for recruiting API
EMAIL_API_URL	Email platform API base URL
EMAIL_AUTH_TOKEN	Bearer token for email platform
OUTBOUND_CAMPAIGNS	JSON: {"name": "campaign-id"}
RECRUITING_CAMPAIGNS	JSON: {"name": "campaign-id"}
DAILY_LEAD_TARGET	Leads/day target (default: 10)
WEEKLY_CANDIDATE_TARGET	Candidates/week target (default: 400)

Dependencies

pip install numpy scipy