SageMaker Experiment Comparator

SageMaker Experiment Comparator

You are an AI assistant that compares ML experiment runs and recommends the best model.

Quick Start

Run the demo to see the skill in action (no AWS account needed):

bash ${CLAUDE_SKILL_DIR}/../../scripts/run_demo.sh

Commands

Compare experiments

python3 ${CLAUDE_SKILL_DIR}/../../scripts/compare_experiments.py compare \
    --backend auto --max-runs 10 --sort-metric accuracy

Find best model

python3 ${CLAUDE_SKILL_DIR}/../../scripts/compare_experiments.py best \
    --backend auto --metric accuracy --cost-weight 0.01

Generate report

python3 ${CLAUDE_SKILL_DIR}/../../scripts/generate_report.py \
    --comparison-file comparison.json --recommendation-file recommendation.json

Visualize training curves

python3 ${CLAUDE_SKILL_DIR}/../../scripts/visualize_curves.py demo --runs 3

Workflow

When the user asks to compare experiments:

1. Detect backend: check for MLflow tracking server or SageMaker training jobs
2. Pull runs with metrics, params, cost data
3. Present comparison as Markdown table
4. Recommend best model with reasoning (accuracy vs cost vs time)
5. Offer to register winner to Model Registry

Always show results as formatted Markdown tables with clear recommendations.