Prompt Template Manager

Prompt Template Manager

Version-control, parameterize, and A/B test LLM prompt templates using Git-native workflows and structured registries.

Activation Triggers

Activate on: "prompt versioning", "prompt template", "A/B test prompts", "prompt registry", "manage prompt variants", "prompt as code", "parameterized prompts", "prompt lifecycle"

NOT for: Writing or optimizing individual prompts (prompt-engineer), fine-tuning dataset preparation (fine-tuning-dataset-curator), or LLM application architecture (ai-engineer)

Quick Start

1. Structure prompts as files — Store each prompt template in prompts/{domain}/{name}.yaml with metadata, variables, and versioned content.
2. Parameterize with variables — Use {{variable}} placeholders for dynamic content; separate static instruction from dynamic context.
3. Version with Git — Each prompt change is a commit. Tags mark production versions. Branches for experiments.
4. A/B test — Deploy multiple variants simultaneously, route traffic by percentage, collect metrics per variant.
5. Promote winners — Evaluate via automated harness or human review, promote winning variant to production tag.

Core Capabilities

Architecture Patterns

Pattern 1: File-Based Prompt Registry

prompts/
├── customer-support/
│   ├── ticket-classifier.yaml      # Active production template
│   ├── ticket-classifier.v2.yaml   # Experiment variant
│   └── response-generator.yaml
├── content/
│   ├── blog-outline.yaml
│   └── social-post.yaml
└── _shared/
    ├── system-safety.yaml          # Reusable system prompt fragments
    └── output-format-json.yaml

# prompts/customer-support/ticket-classifier.yaml
name: ticket-classifier
version: "1.3.0"
model: claude-sonnet-4-20250514
temperature: 0
description: Classify support tickets into categories
variables:
  - name: ticket_text
    type: string
    required: true
  - name: categories
    type: list
    default: [billing, technical, account, other]
includes:
  - _shared/output-format-json.yaml

template: |
  You are a support ticket classifier.

  Classify the following ticket into exactly one category.
  Categories: {{categories | join(", ")}}

  Ticket:
  {{ticket_text}}

  {{> output-format-json}}

tests:
  - input: { ticket_text: "I can't log in to my account" }
    expected_category: "account"
  - input: { ticket_text: "You charged me twice" }
    expected_category: "billing"

Pattern 2: A/B Testing Pipeline

Request ──→ [Router] ──→ Variant A (control, 80%) ──→ [LLM] ──→ Response + Log
                │                                                      │
                └──→ Variant B (experiment, 20%) ──→ [LLM] ──→ Response + Log
                                                                       │
                                                                       ▼
                                                              [Metrics Store]
                                                              - latency
                                                              - token count
                                                              - quality score
                                                              - user feedback
                                                                       │
                                                                       ▼
                                                              [Evaluation]
                                                              Winner → promote to 100%

# A/B routing with metrics
import random, time

class PromptRouter:
    def __init__(self, registry, metrics):
        self.registry = registry
        self.metrics = metrics

    def resolve(self, template_name: str, user_id: str) -> dict:
        variants = self.registry.get_variants(template_name)
        # Deterministic assignment by user_id for consistency
        bucket = hash(f"{user_id}:{template_name}") % 100
        for variant in variants:
            if bucket < variant["traffic_pct"]:
                self.metrics.log("variant_assigned", {
                    "template": template_name,
                    "variant": variant["version"],
                    "user_id": user_id
                })
                return variant
            bucket -= variant["traffic_pct"]
        return variants[0]  # Default to control

Pattern 3: Prompt Composition with Includes

System Prompt = [safety-preamble] + [role-definition] + [output-format]
User Prompt   = [context-injection] + [user-query] + [constraints]

Compose from reusable fragments:
  _shared/safety-preamble.yaml  ──→ "You must not generate harmful content..."
  _shared/json-output.yaml     ──→ "Respond with valid JSON matching this schema..."
  domain/role.yaml              ──→ "You are an expert in {{domain}}..."

Final prompt = render(compose([safety, role, output]), variables)

Anti-Patterns

1. Prompts hardcoded in application code — Changing a prompt requires a code deploy. Store prompts externally with hot-reload capability.
2. No version tracking — "Which prompt was running when quality dropped last Tuesday?" Without versions tied to timestamps, this is unanswerable.
3. A/B testing without statistical significance — Declaring a winner after 50 requests is noise. Require minimum sample sizes (typically 200+ per variant) and p-value thresholds.
4. Monolithic mega-prompts — A 3,000-token prompt that handles every case is fragile. Compose from reusable fragments so changes are isolated.
5. No rollback plan — Every prompt version must be instantly rollbackable by re-tagging the previous version as production.

Quality Checklist

• Prompts stored as files with YAML metadata (not hardcoded in application code)
• Variables parameterized with types and validation (required, default, enum)
• Each prompt has inline test cases for regression testing
• Git tags mark production versions (semantic versioning: major.minor.patch)
• A/B test routing is deterministic per user (no flickering between variants)
• Minimum sample size enforced before declaring A/B test winners
• Metrics collected per variant: latency, tokens, quality score, user feedback
• Reusable fragments extracted to _shared/ for composition
• Rollback possible in under 1 minute (re-tag or config flip)
• Prompt rendering tested: all variables substituted, no raw {{placeholders}} in output