Prompt Evolution System

Prompt Evolution System

Patterns for metrics-driven prompt improvement — externalized storage, A/B testing, quality gates, and auto-rollback.

Three Phases

PromptStore Interface

interface IPromptStore {
  getPrompt(id: string, variables: Record<string, unknown>): Promise<ResolvedPrompt>;
  getVariants(id: string): Promise<PromptVariant[]>;
  recordExecution(id: string, metrics: ExecutionMetrics): Promise<void>;
  promoteVariant(id: string, variantId: string): Promise<void>;
  rollback(id: string, toVersion: string): Promise<void>;
}

// Both LocalPromptStore and CosmosPromptStore implement this interface
// Swap is transparent — controlled by environment config

Metrics Framework

interface ExecutionMetrics {
  promptId: string;
  variantId?: string;
  timestamp: number;
  latency: number;         // ms
  tokenCount: number;
  modelUsed: string;
  qualityScore?: number;   // 0-1, from automated evaluation
  userFeedback?: number;   // 1-5 star rating (if collected)
}

Evolution Lifecycle

Observe → Analyze → Propose → A/B Test → Evaluate → Promote or Rollback

Thresholds

EvolutionEngine Interface

interface IEvolutionEngine {
  analyzePerformance(promptId: string): Promise<PerformanceReport>;
  proposeVariant(promptId: string, direction: EvolutionDirection): Promise<PromptVariant>;
  startABTest(promptId: string, variantId: string): Promise<ABTestConfig>;
  evaluateTest(testId: string): Promise<ABTestResult>;
  promote(promptId: string, variantId: string): Promise<void>;
  rollback(promptId: string): Promise<void>;
}

type EvolutionDirection =
  | 'more_engaging'       // increase user interaction
  | 'more_concise'        // reduce token usage
  | 'more_character'      // strengthen persona voice
  | 'more_accurate'       // improve factual consistency
  | 'better_guidance'     // more helpful without spoiling
  | 'locale_optimize';    // improve non-English prompts

Quality Gate

All evolved prompts must pass quality validation before promotion:

async function qualityGate(
  original: ResolvedPrompt,
  variant: ResolvedPrompt,
): Promise<QualityGateResult> {
  // 1. Generate sample outputs from each prompt
  // 2. Compare quality scores
  // 3. Check for persona drift (if applicable)
  // 4. If drift > threshold: REJECT variant
  // 5. If quality improvement + consistency maintained: APPROVE

  return {
    approved: qualityDelta > 0.05 && driftScore < DRIFT_THRESHOLD,
    qualityDelta,
    driftScore,
    reason: string,
  };
}

Safety Guards

Implementation Checklist

Phase 1: Static

• Define prompt templates in JSON/YAML files
• Implement LocalPromptStore with variable interpolation
• Add basic execution logging (latency, token count)
• Establish baseline metrics for each prompt

Phase 2: Observing

• Migrate to persistent store (Cosmos DB, PostgreSQL, etc.)
• Add comprehensive metrics collection
• Build performance dashboard
• Identify underperforming prompts manually

Phase 3: Evolving

• Implement EvolutionEngine with AI-assisted variant generation
• Add A/B testing infrastructure (traffic splitting, metric comparison)
• Implement statistical significance testing
• Add auto-rollback on quality regression
• Deploy quality gate for all promotions

Activation Patterns