Validation Scoring

Validation Scoring

Pipeline stage: ScoringAgent | Merged from: startup-idea-validation, startup-metrics, startup-expertise

Evidence-based scoring engine for startup ideas. 9-dimension scorecard, 15-domain risk taxonomy, 5-phase validation workflow, stage-appropriate benchmarks, and deterministic scoring formulas. Synthesized from 18 frameworks (HBS, YC, Ash Maurya, Strategyzer, Failory, FI.co).

When to Use

Intake Checklist (Ask First)

• One-sentence idea + target user + job-to-be-done
• Business model: B2B/B2C, SaaS/usage-based/marketplace/services, ACV range
• Geography, constraints (regulated domain, data access, procurement)
• Target outcome: venture-scale, profitable small business, or thesis-driven R&D
• Current evidence: interviews, pilots, pre-sales, traffic, competitors, pricing assumptions

1. 9-Dimension Scorecard

Verdict Thresholds

Deterministic Scoring Formula

overall_score = round(clamp(sum(dimension_score[i] * weight[i] / 100) + bias_correction, 0, 100))

Pipeline implementation weights (7-dimension variant used in scoring-math.ts):

Pipeline verdict thresholds: 75-100 GO | 50-74 CAUTION | 0-49 NO-GO

Factor Status Derivation

Factor scores (1-10) from LLM map to status labels:

Processing Steps

1. Clamp each dimension score to [0, 100]
2. Compute weighted average using dimension weights
3. Apply bias correction (starts at 0, calibrate after 20+ runs)
4. Clamp result to [0, 100], round to nearest integer
5. Derive verdict from threshold table
6. Derive factor statuses from factor scores
7. Build scores_matrix for frontend rendering

Bias Correction

• Starts at 0 (no correction)
• metadata.raw_weighted_average stored for auditing
• After 20+ runs, compare against human baseline and adjust

2. Evidence Grading

A/B/C/D Hierarchy

Signal Strength Hierarchy

verbal (low) < signup (medium) < payment (high) < referral (very high)

An idea is NOT validated until you have payment-level signal.

Triangulation Requirement

• Every claim requires 2+ independent sources
• Especially critical for market sizing and competitor claims
• Strong evidence = behavioral commitment with cost (time, money, switching)
• Weak evidence = opinions and hypotheticals

Counter-Signal Requirement

For every positive finding, require at least one counter-signal:

• Market is large -> but growth is slowing in segment X
• Customers express pain -> but current workarounds are "good enough" for 60%
• No direct competitors -> but adjacent players could enter easily

3. 15-Domain Risk Taxonomy

Full taxonomy in references/risk-taxonomy.md.

Domain Categories

DESIRABILITY (test first for pre-traction startups)
  Problem Risk       Is this a real, painful, frequent problem?
  Customer Risk      Can you identify, reach, and convert the buyer?
  Channel Risk       Is there a viable, scalable acquisition path?
  Value Prop Risk    Is your offering 10x better than alternatives?

VIABILITY (test second)
  Revenue Risk       Will customers pay enough, often enough?
  Financial Risk     Can you sustain burn rate to PMF?
  Market Risk        Is the market large enough and growing?
  Competition Risk   Can you differentiate and defend position?

FEASIBILITY (test third)
  Technical Risk     Can you build it reliably at scale?
  Team Risk          Do you have the right people and skills?
  Operational Risk   Can you deliver consistently post-launch?

EXTERNAL (monitor continuously)
  Legal Risk         IP, compliance, regulatory barriers
  Pivot Risk         Can you adapt if core assumptions fail?
  Reputational Risk  Data privacy, trust, ethics concerns
  Exit Risk          Acquisition readiness, valuation path

Composite Risk Scoring

RISK SCORE = Impact (1-10) x Probability (1-5) = 0-50

Impact:       1-3 = inconvenient  |  4-6 = serious  |  7-10 = fatal
Probability:  1 = very unlikely   |  3 = coin flip   |  5 = very likely

Prioritization Rules

1. Pre-traction -> desirability risks first (Ash Maurya Theory of Constraints)
2. Within domain -> highest risk score first (FI.co scoring)
3. Equal scores -> cheapest-to-test first (Iterative.vc cost optimization)
4. Max 5 active risks in validation queue (cognitive load management)

Theory of Constraints (TOC)

Constraint shifts as you validate:
  Pre-traction  ->  Acquisition (Do people want this?)
  Early traction ->  Activation (Do they get value quickly?)
  Growing        ->  Retention (Do they come back?)
  Scaling        ->  Revenue (Does unit economics work at scale?)

After each validation cycle, re-diagnose the constraint.

4. Validation Ladder

AI/Automation Validation

If the idea depends on AI, validate explicitly:

• Data rights: Can you legally and reliably access required data?
• Reliability: Define success metrics, failure modes, and human fallback
• Cost-to-serve: Model inference + retrieval + human-in-the-loop costs

5. Five-Phase Validation Workflow

ARTICULATE -> ASSESS -> PRIORITIZE -> TEST -> DECIDE
"What do I    "How big    "What's the   "Does evidence  "Go, pivot,
 believe?"     is the      #1 risk?"      support it?"    or kill?"
               risk?"

Phase 1: ARTICULATE -- Write Down Assumptions

Frame every belief as a hypothesis:

"We believe [X] because [evidence]. If wrong, [consequence]."

Separate facts (verified) from assumptions (unverified). Assign confidence: high / medium / low.

Phase 2: ASSESS -- Research the Landscape

Minimum 10-15 web searches across 5 dimensions:

1. Market opportunity (TAM/SAM/growth)
2. Competitive landscape (direct, indirect, adjacent)
3. Problem validation (pain points, behavior, spend)
4. Trends (technology, regulatory, investment)
5. Business model (pricing, unit economics, channels)

Phase 3: PRIORITIZE -- Find the #1 Risk

Score each assumption: Impact (1-10) x Probability (1-5) = 0-50

Rules: desirability first (pre-traction) -> highest score -> cheapest-to-test -> max 5 active.

Phase 4: TEST -- Run the Cheapest Effective Experiment

Pre-register pass/fail thresholds before running. Use signal strength hierarchy to select method.

Phase 5: DECIDE -- Verdict

Score all dimensions, apply evidence quality. Produce decision memo with GO / CONDITIONAL / PIVOT / NO-GO.

6. Health Score Calculation

5 dimensions weighted by startup stage. Produces overall score 0-100.

Dimensions

Stage Weights

Key insight: At idea stage, product and market dominate. As you progress, traction and financial take over.

7. YC Success Factors

What YC Partners Look For

1. Clarity -- Can you explain your startup in one sentence?
2. Traction -- What have you built? What do users say?
3. Insight -- What do you know that others don't?
4. Determination -- What obstacles have you overcome?
5. Speed -- How fast do you move?

Common Mistakes

• Vague problem description
• No customer quotes
• Feature-focused instead of problem-focused
• Unrealistic market size (top-down TAM)
• No unfair advantage articulated

Stage-Appropriate YC Advice

8. Stage-Appropriate Benchmarks

Pre-Seed

Seed

Series A

Series B+

9. Churn Benchmarks by Segment

Monthly Churn

Net Dollar Retention (NDR)

Gross Retention

LTV:CAC Interpretation

10. Efficiency Metrics

Burn Multiple

Burn Multiple = Net Burn / Net New ARR

Magic Number

Magic Number = Net New ARR (quarter) / S&M Spend (prior quarter)

Rule of 40

Rule of 40 = Growth Rate (%) + Profit Margin (%)

Target: > 40% for Series B+ companies.

Quick Ratio

Quick Ratio = (New MRR + Expansion MRR) / (Churned MRR + Contraction MRR)

Core Unit Economics

CAC = Sales & Marketing Spend / New Customers
LTV = ARPU x Gross Margin x (1 / Monthly Churn)
CAC Payback = CAC / (ARPU x Gross Margin)     # in months
Runway = Cash / (Monthly Expenses - Monthly Revenue)
Net New MRR = New MRR + Expansion - Contraction - Churned
NDR = (ARR Start + Expansion - Contraction - Churn) / ARR Start

Runway Planning

11. Cognitive Bias Detection

Six biases that derail startup validation. Full details in references/cognitive-biases.md.

Red Flags in Founder Reasoning

Defense Strategies

1. Pre-registration -- Write pass/fail criteria BEFORE running any experiment
2. Devil's advocate -- For every key assumption, actively argue the opposite
3. Evidence grading -- Grade every data point A/B/C/D
4. Counter-signal requirement -- Every positive finding needs one counter-signal
5. Time-boxing -- No Level 3+ signal in 4 weeks -> formal pivot review; 3+ failed experiments -> mandatory kill/pivot; max 8 weeks from idea to go/no-go

12. Industry Playbook Context

19 industry playbooks available for scoring calibration.

Full list: Query SELECT industry_id, display_name FROM industry_playbooks WHERE is_active = true

Playbook Fields for Scoring Calibration

Lean Canvas Mapping to Risk Domains

13. Top 20 Metrics Quick Reference

Compact lookup for the most-used startup KPIs. Full formulas in Section 10.

14. TypeScript Metric Calculations

Copy-paste-ready functions for edge functions and frontend utilities.

Revenue

function calculateNetNewMRR(new_mrr: number, expansion: number, contraction: number, churned: number): number {
  return new_mrr + expansion - contraction - churned;
}

Unit Economics

function calculateCAC(salesMarketingCost: number, newCustomers: number): number {
  return salesMarketingCost / newCustomers;
}

function calculateLTV(arpu: number, grossMargin: number, monthlyChurn: number): number {
  return arpu * grossMargin * (1 / monthlyChurn);
}

function calculateCACPayback(cac: number, arpu: number, margin: number): number {
  return cac / (arpu * margin); // months
}

Financial Health

function calculateRunway(cashBalance: number, monthlyExpenses: number, monthlyRevenue: number): number {
  const netBurn = monthlyExpenses - monthlyRevenue;
  return netBurn > 0 ? cashBalance / netBurn : Infinity;
}

function calculateQuickRatio(newMRR: number, expansion: number, churned: number, contraction: number): number {
  return (newMRR + expansion) / (churned + contraction);
}

function calculateRuleOf40(growthRate: number, profitMargin: number): number {
  return growthRate + profitMargin;
}

15. Implementation Patterns

TypeScript interfaces and patterns for integrating scoring into edge functions and frontend.

Health Score Interface

// Reference: health-scorer edge function
interface HealthScore {
  overall: number;  // 0-100
  breakdown: {
    product: number;   // Feature completeness, user feedback
    market: number;    // Competition, timing, size
    team: number;      // Completeness, experience
    traction: number;  // Growth rate, engagement
    financial: number; // Runway, unit economics
  };
  recommendations: string[];
}

// Weights by stage (matches Section 6)
const stageWeights = {
  idea: { product: 0.3, market: 0.3, team: 0.2, traction: 0.1, financial: 0.1 },
  seed: { product: 0.25, market: 0.2, team: 0.15, traction: 0.25, financial: 0.15 },
  series_a: { product: 0.2, market: 0.15, team: 0.15, traction: 0.25, financial: 0.25 },
};

IndustryPlaybook Interface

interface IndustryPlaybook {
  industry_id: string;
  display_name: string;
  narrative_arc: string;           // Story structure for pitches
  prompt_context: string;          // AI prompt enhancement
  investor_expectations: object;   // What investors look for
  failure_patterns: string[];      // Common mistakes
  success_stories: object[];       // Examples
  benchmarks: object;              // Industry benchmarks
  terminology: object;             // Industry-specific terms
  gtm_patterns: object;            // Go-to-market strategies
  decision_frameworks: object;     // How to make decisions
  investor_questions: string[];    // FAQ from investors
  warning_signs: string[];         // Red flags
  stage_checklists: object;        // Stage-specific tasks
  slide_emphasis: object;          // Pitch deck guidance
}

Canvas Analysis Interface

// Reference: lean-canvas-agent edge function
interface CanvasAnalysis {
  completeness: number;  // 0-100
  coherence: number;     // 0-100 (do sections align?)
  gaps: string[];        // Missing or weak sections
  suggestions: {
    section: string;
    current: string;
    suggested: string;
    reason: string;
  }[];
}

// Section validation rules
const validationRules = {
  problem: { min_length: 50, requires: ['pain_point', 'frequency'] },
  solution: { must_address: 'problem', avoid: ['feature_list'] },
  unique_value: { formula: 'For [X] who [Y], we [Z] unlike [W]' },
  unfair_advantage: { not_allowed: ['first_mover', 'passion', 'hard_work'] },
};

Injecting Industry Context into AI Prompts

async function buildPromptWithPlaybook(
  basePrompt: string,
  startupId: string
): Promise<string> {
  const { data: startup } = await supabase
    .from('startups')
    .select('industry')
    .eq('id', startupId)
    .single();

  const { data: playbook } = await supabase
    .from('industry_playbooks')
    .select('prompt_context, terminology, benchmarks')
    .eq('industry_id', startup.industry)
    .single();

  if (!playbook) return basePrompt;

  return `${basePrompt}

Industry Context (${startup.industry}):
${playbook.prompt_context}

Key Terminology:
${JSON.stringify(playbook.terminology, null, 2)}

Industry Benchmarks:
${JSON.stringify(playbook.benchmarks, null, 2)}`;
}

Stage-Appropriate Task Generation

async function generateStageTasks(
  startupId: string
): Promise<Task[]> {
  const { data: startup } = await supabase
    .from('startups')
    .select('stage, industry')
    .eq('id', startupId)
    .single();

  const { data: playbook } = await supabase
    .from('industry_playbooks')
    .select('stage_checklists')
    .eq('industry_id', startup.industry)
    .single();

  const checklist = playbook?.stage_checklists?.[startup.stage] || [];

  return checklist.map(item => ({
    title: item.title,
    description: item.description,
    category: item.category,
    priority: item.priority,
    stage_relevant: true,
  }));
}

16. Edge Functions & Database Tables

Edge Functions for Startup Features

Database Tables

Prompt Packs

Pre-built AI workflows for common startup tasks. Each pack is a multi-step prompt sequence.

// Accessing prompt packs in edge functions
const { data: pack } = await supabase
  .from('prompt_packs')
  .select('*')
  .eq('category', 'fundraising')
  .eq('name', 'pitch_deck_outline')
  .single();

for (const step of pack.steps) {
  const result = await executePromptStep(step, context);
}

17. AI Model Selection

18. Implementation Best Practices

When Building Startup Features

• Check if industry playbook exists for user's industry
• Use stage-appropriate metrics and guidance (Section 8)
• Reference YC best practices where applicable (Section 7)
• Include industry-specific terminology from playbook
• Validate against common failure patterns
• Provide actionable recommendations, not just scores

When Accessing Playbooks

• Always filter by is_active = true
• Cache playbook data (update frequency: monthly)
• Fall back to generic advice if industry not found
• Use prompt_context to enhance AI responses

When Implementing Scoring

• Use stage-appropriate weights (Section 6)
• Provide breakdown, not just total score
• Include specific recommendations
• Compare to industry benchmarks where available
• Explain score changes over time

Model-Specific Metrics References

For deeper metrics by business model, see:

• SaaS: saas-metrics.md -- NDR, magic number, burn multiple, investor expectations
• Marketplace / Consumer / B2B: marketplace-consumer-b2b-metrics.md -- GMV, take rate, K-factor, pipeline
• Tracking setup: tracking-best-practices.md -- Infrastructure, cadence, common mistakes

19. References

Reference Files

Scripts

Implementation Files