AI Model Evaluation

AI Model Evaluation

Help PMs systematically evaluate AI models (LLMs, ML APIs, fine-tuned models) for product fit using a structured framework.

Context

You are helping evaluate AI models or vendors for $ARGUMENTS.

Instructions

Phase 0: Context Confirmation

Before proceeding, confirm your understanding of the request:

1. Summarize what you understand from $ARGUMENTS — restate the product, feature, or situation back to the user in 2-3 sentences.
2. Identify gaps — check whether the following are clear (ask if not):
   • What task will the model perform?
   • Who are the end users and what quality bar do they expect?
   • What are the hard constraints (latency, cost, data residency)?
   • Have any candidate models been identified already?
3. Confirm: "Here's my understanding: [summary]. I plan to evaluate candidate models across quality, latency, cost, compliance, and vendor risk dimensions and produce a scored evaluation report. Does this look right, or would you like to adjust anything before I proceed?"

If the user provides additional context, incorporate it before moving to Step 1. If the user confirms, proceed.

Once context is confirmed, proceed to the detailed analysis steps below.

1. Clarify the use case:

   • What task will the model perform? (classification, generation, summarisation, code, multimodal)
   • Who are the end users and what quality bar do they expect?
   • What are the hard constraints? (latency budget, cost ceiling, data residency, compliance requirements)

2. Identify candidate models / vendors:

   • List at least three candidates spanning foundation model APIs (e.g., OpenAI, Anthropic, Google), open-weight models (e.g., Llama, Mistral), and fine-tuned alternatives
   • Note the latest available versions for each

3. Score each candidate on the evaluation matrix (see table below):

   • Rate each dimension 1–5 and explain the rating
   • Weight dimensions according to the user's stated priorities

4. Assess quality benchmarks:

   • Generation tasks: BLEU, ROUGE-L, BERTScore, human evaluation, LLM-as-judge
   • Classification tasks: precision, recall, F1, AUC-ROC on a held-out test set
   • RAG / grounded tasks: hallucination rate, groundedness score, citation accuracy
   • Recommend running a small offline eval (50–200 examples) before committing

5. Model operational requirements:

   • Latency: p50 / p95 / p99 requirements vs. measured API latency; streaming availability
   • Throughput: requests per second needed; rate limits of candidate APIs
   • Context window: max tokens needed for the use case (with headroom for prompt + output)
   • Modality support: text, images, audio, video, tool/function calling, structured outputs

6. Cost modelling:

   • Estimate monthly cost at target request volume: requests/month × avg_tokens × price_per_token
   • Compare input vs. output token pricing; consider caching and batching savings
   • Model cost trajectory as volume scales 10× and 100×

7. Fine-tuning and customisation:

   • Does the model support fine-tuning? (LoRA, full fine-tune, RLHF)
   • What data volume is required? What is the fine-tuning cost and cadence?
   • Evaluate RAG as a lower-cost customisation alternative

8. Data privacy and compliance:

   • GDPR: Is data processed in the EU? Is there a data processing agreement?
   • HIPAA: Is the vendor a BAA signatory?
   • SOC 2 Type II certification status
   • Zero data retention / training opt-out policies
   • Data residency and sovereignty requirements

9. Vendor risk:

   • Deprecation risk: historical model deprecation timeline; migration notice periods
   • Lock-in risk: proprietary API surface vs. OpenAI-compatible endpoints
   • Stability: SLA uptime guarantees, incident history
   • Company viability: funding, revenue, strategic roadmap

10. Decision framework — build vs. API vs. fine-tune:

    • Use API as-is: commodity task, fast time-to-market, no data advantage, low volume
    • Fine-tune base model: domain-specific language/style, consistent format, moderate data available
    • Build custom model: core differentiator, large proprietary dataset, regulatory requirement, extreme cost sensitivity at scale

11. Produce evaluation report:

    • Scoring matrix (see template below)
    • Top recommendation with rationale
    • Risks and mitigations
    • Suggested proof-of-concept scope

Evaluation Scoring Matrix

Think step by step. Save as markdown.