Prompt Performance Tester

Prompt Performance Tester

Model-agnostic prompt benchmarking across 9 providers.

Pass any model ID — provider auto-detected. Compare latency, cost, quality, and consistency across Claude, GPT, Gemini, DeepSeek, Grok, MiniMax, Qwen, Llama, and Mistral.

🚀 Why This Skill?

Problem Statement

Comparing LLM models across providers requires manual testing:

• No systematic way to measure performance across models
• Cost differences are significant but not easily comparable
• Quality varies by use case and provider
• Manual API testing is time-consuming and error-prone

The Solution

Test prompts across any model from any supported provider simultaneously. Get performance metrics and recommendations based on latency, cost, and quality.

Example Cost Comparison

For 10,000 requests/day with average 28 input + 115 output tokens:

• Claude Opus 4.6: ~$30.15/day ($903/month)
• Gemini 2.5 Flash-Lite: ~$0.05/day ($1.50/month)
• DeepSeek Chat: ~$0.14/day ($4.20/month)
• Monthly cost difference (Opus vs Flash-Lite): $901.50

✨ What You Get

Model-Agnostic Multi-Provider Testing

Pass any model ID — provider is auto-detected from the model name prefix. No hardcoded list; new models work without code changes.

Known Pricing (per 1M tokens)

Note: Unlisted models still work — cost calculation returns $0.00 with a warning. Pricing table is for reference only, not a validation gate.

Performance Metrics

Every test measures:

• ⚡ Latency — Response time in milliseconds
• 💰 Cost — Exact API cost per request (input + output tokens)
• 🎯 Quality — Response quality score (0–100)
• 📊 Token Usage — Input and output token counts
• 🔄 Consistency — Variance across multiple test runs
• ❌ Error Tracking — API failures, timeouts, rate limits

Smart Recommendations

Get instant answers to:

• Which model is fastest for your prompt?
• Which is most cost-effective?
• Which produces best quality responses?
• How much can you save by switching providers?

📊 Real-World Example

PROMPT: "Write a professional customer service response about a delayed shipment"

┌─────────────────────────────────────────────────────────────────┐
│ GEMINI 2.5 FLASH-LITE (Google) 💰 MOST AFFORDABLE              │
├─────────────────────────────────────────────────────────────────┤
│ Latency:  523ms                                                 │
│ Cost:     $0.000025                                             │
│ Quality:  65/100                                                │
│ Tokens:   28 in / 87 out                                        │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│ DEEPSEEK CHAT (DeepSeek) 💡 BUDGET PICK                        │
├─────────────────────────────────────────────────────────────────┤
│ Latency:  710ms                                                 │
│ Cost:     $0.000048                                             │
│ Quality:  70/100                                                │
│ Tokens:   28 in / 92 out                                        │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│ CLAUDE HAIKU 4.5 (Anthropic) 🚀 BALANCED PERFORMER             │
├─────────────────────────────────────────────────────────────────┤
│ Latency:  891ms                                                 │
│ Cost:     $0.000145                                             │
│ Quality:  78/100                                                │
│ Tokens:   28 in / 102 out                                       │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│ GPT-5.2 (OpenAI) 💡 EXCELLENT QUALITY                          │
├─────────────────────────────────────────────────────────────────┤
│ Latency:  645ms                                                 │
│ Cost:     $0.000402                                             │
│ Quality:  88/100                                                │
│ Tokens:   28 in / 98 out                                        │
└─────────────────────────────────────────────────────────────────┘

┌─────────────────────────────────────────────────────────────────┐
│ CLAUDE OPUS 4.6 (Anthropic) 🏆 HIGHEST QUALITY                 │
├─────────────────────────────────────────────────────────────────┤
│ Latency:  1,234ms                                               │
│ Cost:     $0.001875                                             │
│ Quality:  94/100                                                │
│ Tokens:   28 in / 125 out                                       │
└─────────────────────────────────────────────────────────────────┘

🎯 RECOMMENDATIONS:
1. Most cost-effective: Gemini 2.5 Flash-Lite ($0.000025/request) — 99.98% cheaper than Opus
2. Budget pick: DeepSeek Chat ($0.000048/request) — strong quality at low cost
3. Best quality: Claude Opus 4.6 (94/100) — state-of-the-art reasoning & analysis
4. Smart pick: Claude Haiku 4.5 ($0.000145/request) — 81% cheaper, 83% quality match
5. Speed + Quality: GPT-5.2 ($0.000402/request) — excellent quality at mid-range cost

💡 Potential monthly savings (10,000 requests/day, 28 input + 115 output tokens avg):
   - Using Gemini 2.5 Flash-Lite vs Opus: $903/month saved ($1.44 vs $904.50)
   - Using DeepSeek Chat vs Opus: $899/month saved ($4.50 vs $904.50)
   - Using Claude Haiku vs Opus: $731/month saved ($173.40 vs $904.50)

Use Cases

Production Deployment

• Evaluate models before production selection
• Compare cost vs quality tradeoffs
• Benchmark API latency across providers

Prompt Development

• Test prompt variations across models
• Measure quality scores consistently
• Compare performance metrics

Cost Analysis

• Analyze LLM API spending by model
• Compare provider pricing structures
• Identify cost-efficient alternatives

Performance Testing

• Measure latency and response times
• Test consistency across multiple runs
• Evaluate quality scores

🚀 Quick Start

1. Subscribe to Skill

Click "Subscribe" on ClawhHub to get access.

2. Set API Keys

Add keys for the providers you want to test:

# Anthropic (Claude models)
export ANTHROPIC_API_KEY="sk-ant-..."

# OpenAI (GPT models)
export OPENAI_API_KEY="sk-..."

# Google (Gemini models)
export GOOGLE_API_KEY="AI..."

# DeepSeek
export DEEPSEEK_API_KEY="..."

# xAI (Grok models)
export XAI_API_KEY="..."

# MiniMax
export MINIMAX_API_KEY="..."

# Alibaba (Qwen models)
export DASHSCOPE_API_KEY="..."

# OpenRouter (Meta Llama models)
export OPENROUTER_API_KEY="..."

# Mistral
export MISTRAL_API_KEY="..."

You only need keys for the providers you plan to test.

3. Install Dependencies

# Install only what you need
pip install anthropic          # Claude
pip install openai             # GPT, DeepSeek, xAI, MiniMax, Qwen, Llama
pip install google-generativeai  # Gemini
pip install mistralai          # Mistral

# Or install everything
pip install anthropic openai google-generativeai mistralai

4. Run Your First Test

Option A: Python

import os
from prompt_performance_tester import PromptPerformanceTester

tester = PromptPerformanceTester()  # reads API keys from environment

results = tester.test_prompt(
    prompt_text="Write a professional email apologizing for a delayed shipment",
    models=[
        "claude-haiku-4-5-20251001",
        "gpt-5.2",
        "gemini-2.5-flash",
        "deepseek-chat",
    ],
    num_runs=3,
    max_tokens=500
)

print(tester.format_results(results))
print(f"🏆 Best quality:  {results.best_model}")
print(f"💰 Cheapest:      {results.cheapest_model}")
print(f"⚡ Fastest:       {results.fastest_model}")

Option B: CLI

# Test across multiple models
prompt-tester test "Your prompt here" \
  --models claude-haiku-4-5-20251001 gpt-5.2 gemini-2.5-flash deepseek-chat \
  --runs 3

# Export results
prompt-tester test "Your prompt here" --export results.json

🔒 Security & Privacy

API Key Safety

• Keys stored in environment variables only — never hardcoded or logged
• Never transmitted to UnisAI servers
• HTTPS encryption for all provider API calls

Data Privacy

• Your prompts are sent only to the AI providers you select for testing
• Each provider has their own data retention policy (see their privacy pages)
• No data stored on UnisAI infrastructure

📚 Technical Details

System Requirements

• Python: 3.9+
• Dependencies: anthropic, openai, google-generativeai, mistralai (install only what you need)
• Platform: macOS, Linux, Windows

Architecture

• Lazy client initialization — SDK clients only loaded for providers actually tested
• Prefix-based routing — PROVIDER_MAP detects provider from model name; no hardcoded whitelist
• OpenAI-compat path — DeepSeek, xAI, MiniMax, Qwen, and OpenRouter all use the openai SDK with a custom base_url
• Pricing table — used for cost calculation only; unknown models get cost=0 with a warning

Metrics Collected

Every test captures:

• Latency: Total response time (ms)
• Cost: Input + output cost based on known pricing (USD)
• Quality: Heuristic response score based on length, completeness (0–100)
• Tokens: Exact input/output token counts per provider
• Consistency: Standard deviation across multiple runs
• Errors: Timeouts, rate limits, API failures

❓ Frequently Asked Questions

Q: Do I need API keys for all 9 providers? A: No. You only need keys for the providers you want to test. If you only test Claude models, you only need ANTHROPIC_API_KEY.

Q: Who pays for the API costs? A: You do. You provide your own API keys and pay each provider directly. This skill has no per-request fees.

Q: How accurate are the cost calculations? A: Costs are calculated from the known pricing table using actual token counts. Models not in the pricing table return $0.00 — the model still runs, the cost just won't be shown.

Q: Can I test models not in the pricing table? A: Yes. Any model whose name starts with a supported prefix will run. Cost will show as $0.00 for unlisted models.

Q: Can I test prompts in non-English languages? A: Yes. All supported providers handle multiple languages.

Q: Can I use this in production/CI/CD? A: Yes. Import PromptPerformanceTester directly from Python or call via CLI.

Q: What if my prompt is very long? A: Set max_tokens appropriately. The skill passes your prompt as-is to each provider's API.

🗺️ Roadmap

✅ Current Release (v1.1.8)

• Model-agnostic architecture — any model ID works via prefix detection
• 9 providers, 20 known models with pricing
• DeepSeek, xAI Grok, MiniMax, Qwen, Meta Llama as first-class providers
• Claude 4.6 series (opus-4-6, sonnet-4-6)
• Lazy client initialization — only loads SDKs for providers actually used
• Fixed UnisAI branding throughout

🚧 Coming Soon (v1.3)

• Batch testing: Test 100+ prompts simultaneously
• Historical tracking: Track model performance over time
• Webhook integrations: Slack, Discord, email notifications

🔮 Future (v1.3+)

• A/B testing framework: Scientific prompt experimentation
• Fine-tuning insights: Which models to fine-tune for your use case
• Custom benchmarks: Create your own evaluation criteria
• Auto-optimization: AI-powered prompt improvement suggestions

📞 Support

• Email: support@unisai.vercel.app
• Website: https://unisai.vercel.app
• Bug Reports: support@unisai.vercel.app

📄 License & Terms

This skill is distributed via ClawhHub under the following terms.

✅ You CAN:

• Use for your own business and projects
• Test prompts for internal applications
• Modify source code for personal use

❌ You CANNOT:

• Redistribute outside ClawhHub registry
• Resell or sublicense
• Use UnisAI trademark without permission

Full Terms: See LICENSE.md

📝 Changelog

[1.1.8] - 2026-02-27

Fixes & Polish

• Bumped version to 1.1.8
• SKILL.md fully rewritten — cleaned up formatting, removed stale content
• Removed old IP watermark reference (PROPRIETARY_SKILL_VEDANT_2024) from docs
• Corrected watermark to PROPRIETARY_SKILL_UNISAI_2026_MULTI_PROVIDER throughout
• Fixed all UnisAI branding (was UniAI in v1.1.0 changelog)
• Updated pricing table to include all 20 known models
• Cleaned up FAQ, Quick Start, and Use Cases sections

[1.1.6] - 2026-02-27

🏗️ Model-Agnostic Architecture

• Provider auto-detected from model name prefix — no hardcoded whitelist
• Any new model works automatically without code changes
• Added DeepSeek, xAI Grok, MiniMax, Qwen, Meta Llama as first-class providers (9 total)
• Updated Claude to 4.6 series (claude-opus-4-6, claude-sonnet-4-6)
• Lazy client initialization — only loads SDKs for providers actually tested
• Unified OpenAI-compat path for DeepSeek, xAI, MiniMax, Qwen, OpenRouter

[1.1.5] - 2026-02-01

🚀 Latest Models Update

• GPT-5.2 Series — Added Instant, Thinking, and Pro variants
• Gemini 2.5 Series — Updated to 2.5 Pro, Flash, and Flash-Lite
• Claude 4.5 pricing updates
• 10 total models across 3 providers

[1.1.0] - 2026-01-15

✨ Major Features

• Multi-provider support — Claude, GPT, Gemini
• Cross-provider cost comparison
• Enhanced recommendations engine
• Rebranded to UnisAI

[1.0.0] - 2024-02-02

Initial Release

• Claude-only prompt testing (Haiku, Sonnet, Opus)
• Performance metrics: latency, cost, quality, consistency
• Basic recommendations engine

Last Updated: February 27, 2026 Current Version: 1.1.8 Status: Active & Maintained

© 2026 UnisAI. All rights reserved.