Prompt Evolution System
Patterns for metrics-driven prompt improvement — externalized storage, A/B testing, quality gates, and auto-rollback.
Three Phases
| Phase |
When |
Backend |
Capability |
| Static |
MVP |
LocalPromptStore (JSON files) |
Fixed prompts, manual edits |
| Observing |
Month 2+ |
CosmosPromptStore or DB |
Metrics collection, no mutations |
| Evolving |
Month 3+ |
CosmosPromptStore or DB |
Auto-propose variants, A/B test, promote |
PromptStore Interface
interface IPromptStore {
getPrompt(id: string, variables: Record<string, unknown>): Promise<ResolvedPrompt>;
getVariants(id: string): Promise<PromptVariant[]>;
recordExecution(id: string, metrics: ExecutionMetrics): Promise<void>;
promoteVariant(id: string, variantId: string): Promise<void>;
rollback(id: string, toVersion: string): Promise<void>;
}
// Both LocalPromptStore and CosmosPromptStore implement this interface
// Swap is transparent — controlled by environment config
Metrics Framework
| Category |
Metrics |
Purpose |
| Engagement |
session_length, actions_per_minute, return_rate |
User interest |
| Quality |
coherence_score, factual_accuracy, response_relevance |
Output quality |
| Persona |
persona_consistency, tone_rating, helpfulness |
Character voice (if applicable) |
| Task |
task_completion_rate, time_to_complete, accuracy |
Core task metrics |
| CX |
error_rate, timeout_rate, retry_count |
Technical quality |
| Cost |
tokens_per_request, cost_per_session, latency_p95 |
Operational efficiency |
interface ExecutionMetrics {
promptId: string;
variantId?: string;
timestamp: number;
latency: number; // ms
tokenCount: number;
modelUsed: string;
qualityScore?: number; // 0-1, from automated evaluation
userFeedback?: number; // 1-5 star rating (if collected)
}
Evolution Lifecycle
Observe → Analyze → Propose → A/B Test → Evaluate → Promote or Rollback
Thresholds
| Executions |
Stage |
Action |
| 0-25 |
Observe |
Collect baseline metrics only |
| 25-50 |
Identify |
Flag underperforming prompts (below median) |
| 50-100 |
Propose |
EvolutionEngine generates improvement variants |
| 100+ |
A/B Test |
Run variant against baseline (50/50 split) |
| 200+ |
Evaluate |
Statistical significance test (p < 0.05) |
| Significant |
Decide |
If variant > baseline by 5%+: promote. Else: discard |
EvolutionEngine Interface
interface IEvolutionEngine {
analyzePerformance(promptId: string): Promise<PerformanceReport>;
proposeVariant(promptId: string, direction: EvolutionDirection): Promise<PromptVariant>;
startABTest(promptId: string, variantId: string): Promise<ABTestConfig>;
evaluateTest(testId: string): Promise<ABTestResult>;
promote(promptId: string, variantId: string): Promise<void>;
rollback(promptId: string): Promise<void>;
}
type EvolutionDirection =
| 'more_engaging' // increase user interaction
| 'more_concise' // reduce token usage
| 'more_character' // strengthen persona voice
| 'more_accurate' // improve factual consistency
| 'better_guidance' // more helpful without spoiling
| 'locale_optimize'; // improve non-English prompts
Quality Gate
All evolved prompts must pass quality validation before promotion:
async function qualityGate(
original: ResolvedPrompt,
variant: ResolvedPrompt,
): Promise<QualityGateResult> {
// 1. Generate sample outputs from each prompt
// 2. Compare quality scores
// 3. Check for persona drift (if applicable)
// 4. If drift > threshold: REJECT variant
// 5. If quality improvement + consistency maintained: APPROVE
return {
approved: qualityDelta > 0.05 && driftScore < DRIFT_THRESHOLD,
qualityDelta,
driftScore,
reason: string,
};
}
Safety Guards
| Guard |
What It Prevents |
| Rate limiter |
Max 1 evolution per prompt per week |
| Rollback trigger |
Quality drop > 10% after promotion → auto-rollback |
| Safety lock |
Evolution cannot modify safety rules |
| System prompt lock |
Evolution cannot modify instruction hierarchy |
| Quality gate |
Evolution cannot drift persona or reduce quality |
Implementation Checklist
Phase 1: Static
Phase 2: Observing
Phase 3: Evolving
Activation Patterns
| Trigger |
Response |
| "prompt evolution", "prompt versioning" |
Full skill activation |
| "A/B testing", "prompt testing" |
Evolution Lifecycle + Thresholds |
| "prompt metrics", "prompt quality" |
Metrics Framework section |
| "prompt rollback", "quality regression" |
Safety Guards section |
| "prompt store", "externalize prompts" |
PromptStore Interface section |
1---2name: prompt-evolution-system-43description: Externalized prompt store, A/B testing, metrics-driven evolution, quality gates, and auto-rollback4tier: standard5applyTo: '**/*prompt*,**/*evolution*,**/*a-b-test*'6metadata:7 source: https://skillmd.com/skills/fabioc-aloha/prompt-evolution-system-48 version: 1.0.09 category: ai-engineering10 tags: [prompt-evolution, a-b-testing, metrics, prompt-store, versioning, rollback, prompt-management]11 activation:12 triggers: [prompt evolution, a/b test, prompt store, prompt versioning, metrics, prompt rollback, evolution engine]13 context: Working on prompt evolution, A/B testing, prompt metrics, or evolution lifecycle14 prerequisites: []15currency: 2026-04-2216---1718# Prompt Evolution System1920Patterns for metrics-driven prompt improvement — externalized storage, A/B testing, quality gates, and auto-rollback.2122---2324## Three Phases2526| Phase | When | Backend | Capability |27|-------|------|---------|-----------|28| **Static** | MVP | `LocalPromptStore` (JSON files) | Fixed prompts, manual edits |29| **Observing** | Month 2+ | `CosmosPromptStore` or DB | Metrics collection, no mutations |30| **Evolving** | Month 3+ | `CosmosPromptStore` or DB | Auto-propose variants, A/B test, promote |3132---3334## PromptStore Interface3536```typescript37interface IPromptStore {38 getPrompt(id: string, variables: Record<string, unknown>): Promise<ResolvedPrompt>;39 getVariants(id: string): Promise<PromptVariant[]>;40 recordExecution(id: string, metrics: ExecutionMetrics): Promise<void>;41 promoteVariant(id: string, variantId: string): Promise<void>;42 rollback(id: string, toVersion: string): Promise<void>;43}4445// Both LocalPromptStore and CosmosPromptStore implement this interface46// Swap is transparent — controlled by environment config47```4849---5051## Metrics Framework5253| Category | Metrics | Purpose |54|----------|---------|---------|55| **Engagement** | session_length, actions_per_minute, return_rate | User interest |56| **Quality** | coherence_score, factual_accuracy, response_relevance | Output quality |57| **Persona** | persona_consistency, tone_rating, helpfulness | Character voice (if applicable) |58| **Task** | task_completion_rate, time_to_complete, accuracy | Core task metrics |59| **CX** | error_rate, timeout_rate, retry_count | Technical quality |60| **Cost** | tokens_per_request, cost_per_session, latency_p95 | Operational efficiency |6162```typescript63interface ExecutionMetrics {64 promptId: string;65 variantId?: string;66 timestamp: number;67 latency: number; // ms68 tokenCount: number;69 modelUsed: string;70 qualityScore?: number; // 0-1, from automated evaluation71 userFeedback?: number; // 1-5 star rating (if collected)72}73```7475---7677## Evolution Lifecycle7879```80Observe → Analyze → Propose → A/B Test → Evaluate → Promote or Rollback81```8283### Thresholds8485| Executions | Stage | Action |86|-----------|-------|--------|87| 0-25 | Observe | Collect baseline metrics only |88| 25-50 | Identify | Flag underperforming prompts (below median) |89| 50-100 | Propose | EvolutionEngine generates improvement variants |90| 100+ | A/B Test | Run variant against baseline (50/50 split) |91| 200+ | Evaluate | Statistical significance test (p < 0.05) |92| Significant | Decide | If variant > baseline by 5%+: promote. Else: discard |9394### EvolutionEngine Interface9596```typescript97interface IEvolutionEngine {98 analyzePerformance(promptId: string): Promise<PerformanceReport>;99 proposeVariant(promptId: string, direction: EvolutionDirection): Promise<PromptVariant>;100 startABTest(promptId: string, variantId: string): Promise<ABTestConfig>;101 evaluateTest(testId: string): Promise<ABTestResult>;102 promote(promptId: string, variantId: string): Promise<void>;103 rollback(promptId: string): Promise<void>;104}105106type EvolutionDirection =107 | 'more_engaging' // increase user interaction108 | 'more_concise' // reduce token usage109 | 'more_character' // strengthen persona voice110 | 'more_accurate' // improve factual consistency111 | 'better_guidance' // more helpful without spoiling112 | 'locale_optimize'; // improve non-English prompts113```114115---116117## Quality Gate118119All evolved prompts must pass quality validation before promotion:120121```typescript122async function qualityGate(123 original: ResolvedPrompt,124 variant: ResolvedPrompt,125): Promise<QualityGateResult> {126 // 1. Generate sample outputs from each prompt127 // 2. Compare quality scores128 // 3. Check for persona drift (if applicable)129 // 4. If drift > threshold: REJECT variant130 // 5. If quality improvement + consistency maintained: APPROVE131132 return {133 approved: qualityDelta > 0.05 && driftScore < DRIFT_THRESHOLD,134 qualityDelta,135 driftScore,136 reason: string,137 };138}139```140141---142143## Safety Guards144145| Guard | What It Prevents |146|-------|-----------------|147| **Rate limiter** | Max 1 evolution per prompt per week |148| **Rollback trigger** | Quality drop > 10% after promotion → auto-rollback |149| **Safety lock** | Evolution cannot modify safety rules |150| **System prompt lock** | Evolution cannot modify instruction hierarchy |151| **Quality gate** | Evolution cannot drift persona or reduce quality |152153---154155## Implementation Checklist156157### Phase 1: Static158159- [ ] Define prompt templates in JSON/YAML files160- [ ] Implement `LocalPromptStore` with variable interpolation161- [ ] Add basic execution logging (latency, token count)162- [ ] Establish baseline metrics for each prompt163164### Phase 2: Observing165166- [ ] Migrate to persistent store (Cosmos DB, PostgreSQL, etc.)167- [ ] Add comprehensive metrics collection168- [ ] Build performance dashboard169- [ ] Identify underperforming prompts manually170171### Phase 3: Evolving172173- [ ] Implement `EvolutionEngine` with AI-assisted variant generation174- [ ] Add A/B testing infrastructure (traffic splitting, metric comparison)175- [ ] Implement statistical significance testing176- [ ] Add auto-rollback on quality regression177- [ ] Deploy quality gate for all promotions178179---180181## Activation Patterns182183| Trigger | Response |184|---------|----------|185| "prompt evolution", "prompt versioning" | Full skill activation |186| "A/B testing", "prompt testing" | Evolution Lifecycle + Thresholds |187| "prompt metrics", "prompt quality" | Metrics Framework section |188| "prompt rollback", "quality regression" | Safety Guards section |189| "prompt store", "externalize prompts" | PromptStore Interface section |