Test Analysis

Test Analysis

Analyze test results and recommend actions.

Process

Step 1: Input Performance Data

Gather metrics:

• CPL (Cost Per Lead)
• CPA (Cost Per Acquisition)
• CTR (Click-Through Rate)
• CVR (Conversion Rate)
• Spend per creative
• Initiate checkouts (leading indicator)
• Time period data

Step 2: Create Heatmap Visualization

For 2x2 or 2x2x3 Tests:

         | Ad Text 1 | Ad Text 2 |
---------|-----------|-----------|
Headline1|   CPA $X  |   CPA $X  |
Headline2|   CPA $X  |   CPA $X  |

Color Coding:

• Green: Below target CPA
• Yellow: At target CPA
• Red: Above target CPA

For Avatar/Image Tests:

         | Image 1 | Image 2 | Image 3 | ...
---------|---------|---------|---------|----
Avatar 1 |  $CPA   |  $CPA   |  $CPA   |
Avatar 2 |  $CPA   |  $CPA   |  $CPA   |
...

Step 3: Identify Statistical Winners

Winner Criteria (Jason K):

• Doesn't lose more than 1x in 3 days
• Doesn't lose more than 2x in 7 days
• Consistent performance over time

Statistical Confidence:

• Minimum spend: 1.5-2x target CPA
• Minimum conversions: 10+ for confidence
• Look at trends, not single days

Leading Indicators:

• Initiate checkout CPA = ~1/3 of purchase CPA
• High CTR + low CVR = lander issue
• Low CTR + any CVR = creative issue

Step 4: Categorize Results

SCALE - Move to scaling CBO

• Consistent winner over 3+ days
• Below target CPA
• Good volume potential

ITERATE - Create variations

• Shows promise but inconsistent
• Close to target CPA
• Clear element working (hook, angle, etc.)

KILL - Stop spending

• Consistently above target
• No signs of improvement
• Clear loser after sufficient spend

TEST MORE - Needs more data

• Insufficient spend for decision
• Mixed signals
• New variable to isolate

Step 5: Output Analysis Report

## TEST ANALYSIS: [Test Name]
Period: [Date range]
Total Spend: $[Amount]
Target CPA: $[Amount]

---

### HEATMAP: [Test Type]

[Visual heatmap grid]

---

### PERFORMANCE SUMMARY

| Creative | Spend | Leads | CPA | CTR | CVR | Status |
|----------|-------|-------|-----|-----|-----|--------|
| Ad 1     | $X    | X     | $X  | X%  | X%  | SCALE  |
| Ad 2     | $X    | X     | $X  | X%  | X%  | ITERATE|
| Ad 3     | $X    | X     | $X  | X%  | X%  | KILL   |

---

### WINNERS (Move to Scale CBO)

**Ad 1: [Name/Description]**
- CPA: $X (X% below target)
- Key elements: [What's working]
- Volume potential: [Assessment]

---

### ITERATE (Create Variations)

**Ad 2: [Name/Description]**
- CPA: $X (X% above target)
- Promising elements: [What's working]
- Issues: [What to fix]
- Next test: [Specific variation to try]

---

### KILL (Stop Immediately)

**Ad 3: [Name/Description]**
- CPA: $X (X% above target)
- Why it failed: [Analysis]
- Learnings: [What to avoid]

---

### PATTERN ANALYSIS

**What's Working:**
- [Pattern 1]
- [Pattern 2]

**What's Not Working:**
- [Pattern 1]
- [Pattern 2]

**Hypothesis for Next Test:**
- [Based on data, test this next]

---

### RECOMMENDATIONS

**Immediate Actions:**
1. Scale [Ad X] to CBO
2. Kill [Ad Y, Z]
3. Create iterations of [Ad A]

**Next Test Phase:**
- Test type: [Description]
- Variables: [What to test]
- Budget: $[Amount]
- Timeline: [Duration]

**Funnel Optimization:**
- [If lander issues identified]
- [If offer issues identified]

Analysis Framework

Metric Priorities:

1. CPA/CPL (primary)
2. CVR (funnel health)
3. CTR (creative appeal)
4. Initiate checkout (leading indicator)

Time Analysis:

• Day-over-day trends
• Day-of-week patterns
• Hour-of-day patterns (for day-parting)

Diagnostic Questions:

• High CTR, low CVR → Lander problem
• Low CTR → Creative problem
• Good metrics, no scale → Audience saturation
• Inconsistent → Need more data or bid caps

Source: Jason K (heatmap method), Meta-CastovsJasonK