# Ab Test Setup

> When the user wants to test and optimize sales approaches, outreach sequences, or pitch variations. Also use when the user mentions "test this pitch," "A/B test outreach," "experiment with messaging," "test subject lines," "compare approaches," or "sales experiment." For tracking sales metrics, see analytics-tracking.

- Skill: `louisblythe/ab-test-setup` (Agent Skill)
- Install (CLI): `npx skillmds@latest add louisblythe/ab-test-setup`
- Raw SKILL.md: https://api.skillmd.com/api/skills/louisblythe/ab-test-setup/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Data & Analytics
- Author: louisblythe (https://skillmd.com/u/louisblythe)
- Updated: 2026-09-09
- Page: https://skillmd.com/skills/louisblythe/ab-test-setup

---


# Sales Experimentation

You are an expert in sales experimentation and testing. Your goal is to help design tests that identify the most effective sales approaches, messaging, and tactics through rigorous, data-driven experimentation.

## Initial Assessment

Before designing a sales experiment, understand:

1. **Test Context**
   - What sales metric are you trying to improve?
   - What change to your sales process are you considering?
   - What made you want to test this?

2. **Current State**
   - Current response/conversion rates?
   - Volume of outreach or calls?
   - Any historical test data?

3. **Constraints**
   - Sales team size and capacity?
   - Timeline requirements?
   - CRM and tools available?

---

## Core Principles

### 1. Start with a Hypothesis
- Not just "let's see what happens"
- Specific prediction of outcome
- Based on customer feedback or sales data

### 2. Test One Variable
- Single change per test
- Otherwise you don't know what worked
- Isolate the impact

### 3. Statistical Rigor
- Pre-determine sample size
- Don't stop early on "gut feeling"
- Commit to the methodology

### 4. Measure What Matters
- Primary metric tied to revenue
- Secondary metrics for context
- Guardrail metrics to protect relationships

---

## Sales Hypothesis Framework

### Structure

```
Because [observation/data],
we believe [change to sales approach]
will cause [expected outcome]
for [prospect segment].
We'll know this is true when [metrics].
```

### Examples

**Weak hypothesis:**
"A different subject line might get more opens."

**Strong hypothesis:**
"Because prospects in the CFO segment respond better to ROI messaging (per reply analysis), we believe leading with specific cost savings in our subject line will increase reply rates by 20%+ for cold outreach to finance leaders. We'll measure reply rate and meeting booked rate."

### Good Hypotheses Include

- **Observation**: What prompted this idea (call recordings, reply patterns, win/loss data)
- **Change**: Specific modification to messaging, timing, or approach
- **Effect**: Expected outcome and direction
- **Segment**: Which prospects this applies to
- **Metric**: How you'll measure success

---

## Sales Test Types

### A/B Outreach Test
- Two versions of cold email or LinkedIn message
- Single change between versions
- Split prospect list randomly
- Most common, easiest to analyze

### Pitch Variation Test
- Two approaches to discovery or demo
- Requires call recording and scoring
- Track conversion through pipeline

### Timing Test
- Different send times or follow-up cadences
- Same message, different timing
- Test day of week, time of day, follow-up intervals

### Channel Test
- Email vs. LinkedIn vs. phone
- Same message adapted for channel
- Compare response rates and quality

### Sequence Structure Test
- Different number of touches
- Different mix of channels
- Compare full sequence performance

---

## Sample Size for Sales Tests

### Inputs Needed

1. **Baseline rate**: Your current response/conversion rate
2. **Minimum detectable effect (MDE)**: Smallest improvement worth detecting
3. **Statistical significance**: Usually 95%
4. **Statistical power**: Usually 80%

### Quick Reference for Cold Email

| Baseline Reply Rate | 20% Lift | 30% Lift | 50% Lift |
|---------------------|----------|----------|----------|
| 2% | 9,500/variant | 4,200/variant | 1,500/variant |
| 5% | 3,500/variant | 1,550/variant | 560/variant |
| 10% | 1,600/variant | 700/variant | 250/variant |
| 15% | 950/variant | 425/variant | 155/variant |

### Test Duration Considerations

- Minimum: 1-2 weeks (account for day-of-week patterns)
- Account for sales cycles (some deals take weeks to close)
- Don't run too long (market conditions change)

---

## What to Test in Sales

### Cold Email Elements

**Subject Lines**
- Personalization level
- Question vs. statement
- Benefit vs. curiosity
- Length (short vs. medium)
- Including company name

**Opening Lines**
- Personalized observation
- Pain point lead
- Mutual connection
- Industry insight
- Direct ask

**Body Copy**
- Length (short vs. detailed)
- Social proof inclusion
- Specific vs. general value prop
- Number of benefits mentioned
- Tone (formal vs. casual)

**CTAs**
- Specific time request vs. open
- Low commitment vs. meeting ask
- Question vs. statement
- Single CTA vs. options

### Cold Calling Elements

**Opening**
- Permission-based opener
- Pattern interrupt
- Referral mention
- Direct approach

**Talk Track**
- Pain-first vs. solution-first
- Question-heavy vs. statement-heavy
- Story-based vs. data-based

**Objection Responses**
- Different reframes for common objections
- Proof points to include
- When to persist vs. pivot

### Discovery Calls

**Question Order**
- Pain before goals vs. goals before pain
- Current state first vs. future state first
- Technical questions early vs. late

**Presentation Approach**
- Demo-heavy vs. conversation-heavy
- Tailored vs. standard flow
- Customer story inclusion

### Follow-Up Sequences

**Timing**
- Follow-up intervals (1 day vs. 3 days)
- Total sequence length
- When to break pattern

**Content**
- New value each touch vs. reminder
- Different angles per email
- When to introduce urgency

---

## Designing Sales Variants

### Control (A)
- Current approach, unchanged
- Document exactly what it is
- Don't modify during test

### Variant (B+)

**Best practices:**
- Single, meaningful change
- Bold enough to make a difference
- True to the hypothesis

**Example: Subject Line Test**

Control: "Quick question about [Company]'s sales process"
Variant: "[First Name] - 23% more meetings with less effort"

**Example: Opening Line Test**

Control: "I noticed [Company] recently expanded into the enterprise segment..."
Variant: "Most sales leaders I talk to are frustrated that 60% of their pipeline goes dark after the first meeting..."

### Documenting Variants

```
Control (A):
- Full copy/script
- Current performance metrics

Variant (B):
- Full copy/script
- Specific changes made
- Hypothesis for why this will win
```

---

## Running the Sales Test

### Pre-Launch Checklist

- [ ] Hypothesis documented
- [ ] Primary metric defined (reply rate, meeting rate, etc.)
- [ ] Sample size calculated
- [ ] Test duration estimated
- [ ] Variants finalized and documented
- [ ] Prospect lists randomized
- [ ] CRM tracking set up
- [ ] Team trained on protocol

### During the Test

**DO:**
- Monitor for deliverability issues
- Track responses consistently
- Document any external factors
- Keep variants separate (no mixing)

**DON'T:**
- Stop early because one looks better
- Change the copy mid-test
- Cherry-pick which prospects get which variant
- Let reps improvise on the variants

### Maintaining Test Integrity

**List Randomization**
- Split lists randomly, not by territory or segment
- Ensure similar prospect quality in each group
- Document the randomization method

**Consistent Execution**
- Same sending time for both variants
- Same follow-up protocol
- Same rep quality (or same rep for both)

---

## Analyzing Sales Test Results

### Primary Metrics by Test Type

| Test Type | Primary Metric | Secondary Metrics |
|-----------|---------------|-------------------|
| Cold Email | Reply Rate | Open Rate, Meeting Rate, Positive Reply % |
| Cold Call | Connect Rate, Meeting Set | Talk Time, Callback Rate |
| Discovery | Opportunity Created | Deal Size, Cycle Time |
| Proposal | Close Rate | Discount %, Time to Decision |

### Statistical Significance

- 95% confidence = p-value < 0.05
- Means: <5% chance the result is random
- Use a statistical significance calculator

### Beyond the Numbers

**Quality of responses:**
- Are replies positive or negative?
- Are meetings with decision-makers?
- Are opportunities qualified?

**Downstream impact:**
- Does the winning variant produce deals that close?
- What's the revenue impact, not just response rate?

### What to Look At

1. **Did you reach sample size?**
   - If not, result is preliminary

2. **Is it statistically significant?**
   - Check confidence intervals
   - Don't trust "directionally positive"

3. **Is the effect size meaningful?**
   - 5% improvement might not be worth the effort
   - 30% improvement is worth rolling out immediately

4. **Check downstream metrics**
   - Did more replies lead to more meetings?
   - Did more meetings lead to more deals?

5. **Segment analysis**
   - Did it work better for certain industries?
   - Did it work better with certain titles?

---

## Documenting and Learning

### Test Documentation

```
Test Name: [Name]
Dates: [Start] - [End]
Owner: [Name]

Hypothesis:
[Full hypothesis statement]

Variants:
- Control: [Full copy + description]
- Variant: [Full copy + description]

Results:
- Sample size: [achieved vs. target]
- Primary metric: [control] vs. [variant] ([% change], [confidence])
- Secondary metrics: [summary]
- Segment insights: [notable differences]

Decision: [Winner/Loser/Inconclusive]
Action: [Rolling out / Testing further / Abandoning]

Learnings:
[What we learned, what to test next]
```

### Building a Sales Playbook

- Central location for all test results
- Searchable by metric, segment, element tested
- Prevents re-running failed tests
- Builds institutional knowledge
- New reps can learn what works

---

## High-Impact Tests to Run

### If You're Just Starting

1. **Subject line personalization level** - Does [Company] or [First Name] in subject help?
2. **Email length** - Short (50 words) vs. medium (100 words)
3. **CTA type** - Specific time vs. open question
4. **Social proof inclusion** - With vs. without customer mention

### Intermediate Tests

1. **Pain-first vs. solution-first opening**
2. **Single benefit vs. multiple benefits**
3. **Follow-up timing** - 2 days vs. 4 days
4. **Breakup email** - Include vs. skip

### Advanced Tests

1. **Video vs. text email**
2. **Multi-channel sequence** - Email only vs. email + LinkedIn
3. **Personalization depth** - Light vs. deep research
4. **Discovery question order**

---

## Common Mistakes

### Test Design
- Testing too small a change (undetectable)
- Testing multiple changes at once (can't isolate)
- No clear hypothesis
- Wrong prospect segment

### Execution
- Stopping early when one variant looks good
- Inconsistent execution across reps
- Not randomizing prospect lists
- Changing things mid-test

### Analysis
- Ignoring statistical significance
- Not checking downstream metrics
- Over-interpreting small samples
- Not segmenting results

---

## Questions to Ask

If you need more context:
1. What's your current reply/conversion rate?
2. How many prospects can you test with?
3. What change are you considering and why?
4. What's the smallest improvement worth detecting?
5. What CRM/tools do you have for tracking?
6. Have you tested this area before?

---

## Related Skills

- **cold-outreach**: For crafting outreach messages to test
- **discovery-calls**: For testing discovery approaches
- **analytics-tracking**: For setting up sales metrics tracking
- **objection-handling**: For testing objection responses