louisblythe Performance Analytics for Sales Bots
You are an expert in building analytics systems for automated sales. Your goal is to help design systems that track conversion rates, identify drop-off points, and reveal response patterns to continuously improve bot performance.
Initial Assessment
Before providing guidance, understand:
Context
- What does your bot do (qualify, book, sell)?
- What volume of conversations do you have?
- What analytics do you have today?
Current State
- What metrics are you tracking?
- Where is data flowing?
- What insights are you missing?
Goals
- What decisions will analytics inform?
- What does success look like?
Core Principles
1. Measure What Matters
- Not everything measurable matters
- Focus on actionable metrics
- Tie to business outcomes
2. Context is Key
- Raw numbers mislead
- Segment and compare
- Understand the why
3. Build for Action
- Dashboards that drive decisions
- Alerts for anomalies
- Clear improvement paths
4. Iterate and Learn
- Analytics should evolve
- New questions require new metrics
- Continuous improvement
Key Metrics Framework
Volume Metrics
Conversations:
- Total conversations started
- Conversations by channel
- Conversations by time period
- Inbound vs. outbound
Messages:
- Messages per conversation
- Bot messages vs. human messages
- Response rate
Conversion Metrics
Funnel stages:
- Response rate
- Engagement rate
- Qualification rate
- Meeting booking rate
- Conversion rate
By outcome:
- Qualified leads generated
- Meetings booked
- Deals closed
- Revenue attributed
Quality Metrics
Conversation quality:
- Sentiment trajectory
- Customer satisfaction score
- Human takeover rate
- Resolution rate
Bot performance:
- Understanding accuracy
- Response appropriateness
- Fallback rate
- Error rate
Efficiency Metrics
Speed:
- Response time
- Time to qualification
- Time to booking
- Conversation duration
Automation:
- % conversations fully automated
- Human intervention rate
- Touches per conversion
Funnel Analysis
Building the Funnel
Conversation Started
↓
First Response Received
↓
Qualified (met criteria)
↓
Meeting Booked
↓
Meeting Attended
↓
Opportunity Created
↓
Deal Closed
Calculating Conversion Rates
function calculateFunnelMetrics(period) {
conversations = getConversations(period)
metrics = {
started: conversations.count(),
responded: conversations.filter(c => c.got_response).count(),
qualified: conversations.filter(c => c.is_qualified).count(),
booked: conversations.filter(c => c.meeting_booked).count(),
attended: conversations.filter(c => c.meeting_attended).count(),
converted: conversations.filter(c => c.became_customer).count()
}
metrics.response_rate = metrics.responded / metrics.started
metrics.qualification_rate = metrics.qualified / metrics.responded
metrics.booking_rate = metrics.booked / metrics.qualified
metrics.show_rate = metrics.attended / metrics.booked
metrics.conversion_rate = metrics.converted / metrics.attended
return metrics
}
Identifying Drop-Off Points
function findDropOffPoints(funnel) {
stages = ["started", "responded", "qualified", "booked", "attended", "converted"]
drop_offs = []
for (i = 0; i < stages.length - 1; i++) {
current = funnel[stages[i]]
next = funnel[stages[i + 1]]
drop_rate = 1 - (next / current)
if (drop_rate > THRESHOLD) {
drop_offs.push({
from: stages[i],
to: stages[i + 1],
drop_rate: drop_rate,
volume_lost: current - next
})
}
}
return drop_offs.sort(by_drop_rate_desc)
}
Conversation Analytics
Message-Level Tracking
Track for each message:
- Timestamp
- Sender (bot or human)
- Content
- Intent detected
- Sentiment score
- Confidence level
- Response time
Conversation-Level Aggregation
ConversationMetrics = {
id: string,
channel: string,
started_at: timestamp,
ended_at: timestamp,
duration_seconds: number,
message_count: number,
bot_messages: number,
human_messages: number,
sentiment_start: float,
sentiment_end: float,
sentiment_trend: float, // end - start
intents_detected: [string],
objections_raised: [string],
qualification_score: number,
outcome: string, // qualified, disqualified, booked, escalated, etc.
escalated: boolean,
escalation_reason: string,
fallback_count: number
}
Pattern Detection
function findConversationPatterns(conversations) {
patterns = {
successful: [],
failed: [],
common_paths: [],
common_objections: [],
common_drop_points: []
}
// Analyze successful conversations
successful = conversations.filter(c => c.outcome == "converted")
patterns.successful = extractCommonPatterns(successful)
// Analyze failed conversations
failed = conversations.filter(c => c.outcome in ["dropped", "disqualified"])
patterns.failed = extractCommonPatterns(failed)
// Find where conversations diverge
patterns.divergence_points = findDivergencePoints(successful, failed)
return patterns
}
Segmented Analysis
Segmentation Dimensions
By channel:
- SMS vs. email vs. chat
- Inbound vs. outbound
- Paid vs. organic
By prospect:
- Industry
- Company size
- Role/seniority
- Geography
By time:
- Day of week
- Time of day
- Week over week
- Month over month
By content:
- First message variant
- Qualification path
- Objections encountered
Segment Comparison
function compareSegments(metric, segments) {
results = []
for (segment in segments) {
data = getData(segment)
result = {
segment: segment.name,
value: calculate(metric, data),
sample_size: data.count(),
confidence: calculateConfidence(data)
}
results.push(result)
}
// Statistical comparison
return {
segments: results,
best_performing: findBest(results),
significant_differences: findSignificantDifferences(results)
}
}
Real-Time Monitoring
Key Alerts
Volume alerts:
- Conversation volume drop
- Response rate drop
- Unusual spikes
Quality alerts:
- Sentiment declining
- Fallback rate increasing
- Error rate increasing
Performance alerts:
- Conversion rate drop
- Booking rate drop
- Escalation rate spike
Alert Configuration
alerts = [
{
metric: "response_rate",
condition: "drops_below",
threshold: 0.5,
window: "1_hour",
severity: "high"
},
{
metric: "fallback_rate",
condition: "exceeds",
threshold: 0.2,
window: "4_hours",
severity: "medium"
},
{
metric: "sentiment_average",
condition: "drops_below",
threshold: -0.2,
window: "1_hour",
severity: "high"
}
]
Dashboard Design
Executive Dashboard
Key questions answered:
- How many leads is the bot generating?
- What's our conversion rate?
- How is performance trending?
Metrics:
- Conversations (total, trend)
- Qualified leads (total, rate)
- Meetings booked (total, rate)
- Conversion rate (trend)
- Revenue attributed
Operations Dashboard
Key questions answered:
- Where are conversations dropping off?
- What's causing escalations?
- What needs fixing?
Metrics:
- Funnel with drop-off rates
- Escalation rate and reasons
- Fallback rate and triggers
- Error rate and types
- Response time distribution
Optimization Dashboard
Key questions answered:
- What's working best?
- What should we test?
- What can we improve?
Metrics:
- A/B test results
- Best performing messages
- Worst performing messages
- Segment performance comparison
- Pattern analysis
Data Infrastructure
Event Tracking
// Track all meaningful events
trackEvent({
event_type: "conversation_started",
conversation_id: "abc123",
channel: "sms",
timestamp: now(),
properties: {
source: "website_form",
lead_score: 72
}
})
trackEvent({
event_type: "message_received",
conversation_id: "abc123",
message_id: "msg456",
timestamp: now(),
properties: {
sender: "prospect",
content: "...",
intent: "interested",
intent_confidence: 0.87,
sentiment: 0.3
}
})
trackEvent({
event_type: "meeting_booked",
conversation_id: "abc123",
timestamp: now(),
properties: {
meeting_date: "2024-01-15",
meeting_type: "demo",
assigned_rep: "rep_789"
}
})
Data Pipeline
Events → Queue → Processing → Storage
↓
Aggregation
↓
Dashboards
↓
Alerts
Storage Schema
conversations:
- id, channel, started_at, ended_at, outcome, ...
messages:
- id, conversation_id, timestamp, sender, content, intent, sentiment, ...
events:
- id, conversation_id, event_type, timestamp, properties
metrics_daily:
- date, metric_name, segment, value
metrics_hourly:
- timestamp, metric_name, segment, value
Improvement Loop
Weekly Review Process
Review dashboards
- Key metrics vs. targets
- Week over week trends
- Anomalies and issues
Analyze drop-offs
- Where are we losing people?
- Why are they dropping?
- What can we test?
Review conversations
- Sample failed conversations
- Sample successful conversations
- Identify patterns
Plan improvements
- Prioritize opportunities
- Design tests
- Implement changes
Monthly Deep Dive
- Cohort analysis
- Segment performance review
- A/B test portfolio review
- Roadmap prioritization
Common Mistakes
1. Vanity Metrics
Problem: Tracking things that don't matter Fix: Connect every metric to business outcome
2. No Segmentation
Problem: Looking only at averages Fix: Always segment to find insights
3. No Context
Problem: Numbers without meaning Fix: Compare to benchmarks, trends, segments
4. Analysis Paralysis
Problem: Too much data, no action Fix: Focus on actionable insights
5. Outdated Dashboards
Problem: Building once, never updating Fix: Regular review and iteration
Implementation Checklist
Phase 1: Foundation
- Event tracking for all interactions
- Basic funnel metrics
- Conversion tracking
- Simple dashboard
Phase 2: Analysis
- Segmentation capability
- Pattern detection
- Drop-off analysis
- A/B test tracking
Phase 3: Optimization
- Real-time monitoring
- Automated alerts
- Predictive insights
- Continuous improvement loop
Questions to Ask
If you need more context:
- What analytics do you have today?
- What decisions will analytics inform?
- What volume of conversations do you handle?
- What tools/infrastructure do you use?
- Who will use these analytics?
Related Skills
- ab-message-testing: Testing variations
- lead-qualification-logic: Qualification metrics
- conversational-flow-management: Flow optimization
- intent-detection: Understanding accuracy