Mkt Experiment

Growth Engine

Autonomous growth experimentation framework based on Karpathy's autoresearch pattern applied to marketing. Creates experiments with hypotheses, logs data points, runs statistical analysis (bootstrap CI + Mann-Whitney U), auto-promotes winners to a living playbook, and suggests next experiments. Supports batch mode (up to 10 variants simultaneously).

Usage

Use this skill when:

• Creating or managing A/B or multivariate experiments for any marketing channel
• Logging experiment data points after content is published or campaigns run
• Scoring experiments to determine statistical winners
• Checking the playbook for proven best practices before creating new content
• Generating weekly scorecards across all channels
• Monitoring campaign pacing and health

Do NOT use for:

• One-off content creation (use the playbook output as input, but don't run the engine)
• Non-experiment analytics or reporting
• Campaign setup in external platforms (this tracks experiments, not campaign config)

Commands

Create an experiment

python3 experiment-engine.py create \
  --agent <agent_name> \
  --hypothesis "What you expect to happen" \
  --variable "<variable_name>" \
  --variants '["variant_a", "variant_b"]' \
  --metric "<primary_metric>" \
  --cycle-hours 24

Add --batch-mode for 3-10 variant tests. Add --min-samples N to override auto-detection.

Log a data point

python3 experiment-engine.py log \
  --agent <agent_name> \
  --experiment-id <EXP-ID> \
  --variant "<variant_name>" \
  --metrics '{"metric_name": value}'

Score an experiment

python3 experiment-engine.py score --agent <agent_name> --experiment-id <EXP-ID>

Statuses: running → trending → keep (winner) or discard (loser)

Winners auto-promote to the playbook. Requires p < 0.05 AND ≥ 15% lift.

List experiments

python3 experiment-engine.py list --agent <agent_name> [--status running|trending|keep|discard]

Check the playbook

python3 experiment-engine.py playbook --agent <agent_name>

Always check the playbook before creating new content to apply proven best practices.

Suggest next experiments

python3 experiment-engine.py suggest --agent <agent_name>

Generate weekly scorecard

python3 autogrowth-weekly-scorecard.py [--weeks N] [--output file.md]

Check campaign pacing

python3 pacing-alert.py [--json]

Exit code 0 = on pace, 1 = alerts present.

Workflow

1. Before creating content: playbook → apply proven rules
2. When publishing: log → record which variant was used and its metrics
3. Periodically: score → check if experiments have reached statistical significance
4. Weekly: autogrowth-weekly-scorecard.py → review all channels
5. After completing experiments: suggest → pick the next variable to test

Configuration

Required Environment Variables

Optional Tuning

Pacing Alert Variables

Dependencies

pip install numpy scipy