majiayu000 GPT Lab
Benchmark and compare small GPTs trained by /create-gpt against prompted alternatives.
Answers the key question: "Is fine-tuning worth it for this task?"
Quick Start
cd .pi/skills/gpt-lab
# Benchmark multiple models on a task
./run.sh benchmark --task qra-validator --models "qwen2.5-0.5b,qwen2.5-1.5b"
# Find the smallest model meeting a threshold
./run.sh find-minimum --task qra-validator --threshold 0.85
# Compare fine-tuned vs prompted
./run.sh compare --task qra-validator \
--finetuned ../create-gpt/models/qra-validator/model.gguf \
--prompted deepseek-v3.2
# Profile a single model
./run.sh profile --model ../create-gpt/models/qra-validator/model.gguf --samples 100
# Generate report
./run.sh report --task qra-validator --format markdown
Commands
./run.sh benchmark --task NAME --models "model1,model2,..."
./run.sh compare --task NAME --finetuned PATH --prompted MODEL_NAME
./run.sh find-minimum --task NAME --threshold FLOAT
./run.sh profile --model PATH --samples N
./run.sh report --task NAME [--format markdown|json]
./run.sh history --task NAME
Fine-Tuned vs Prompted Verdict
accuracy delta < -5% → NOT_WORTH_IT
accuracy delta >= -2% AND speedup >= 5x → WORTH_IT
otherwise → MARGINAL
Integration
/create-gpt: Trains the models that this skill benchmarks/scillm: Provides prompted baseline via Chutes API/prompt-lab: find-minimum pattern adapted from this skill/classifier-lab: Benchmark engine pattern adapted from this skill