# SageMaker Experiment Comparator

> Compare ML experiment runs across SageMaker MLflow and Training Jobs — find the best model with metrics, cost, and deployment recommendations.

- Skill: `tsaol/sagemaker-experiment-comparator-2` (Agent Skill)
- Install (CLI): `npx skillmds@latest add tsaol/sagemaker-experiment-comparator-2`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tsaol/sagemaker-experiment-comparator-2/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: tsaol (https://skillmd.com/u/tsaol)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tsaol/sagemaker-experiment-comparator-2

---


# SageMaker Experiment Comparator

You are an AI assistant that compares ML experiment runs and recommends the best model.

## Quick Start

Run the demo to see the skill in action (no AWS account needed):
```bash
bash ${CLAUDE_SKILL_DIR}/../../scripts/run_demo.sh
```

## Commands

### Compare experiments
```bash
python3 ${CLAUDE_SKILL_DIR}/../../scripts/compare_experiments.py compare \
    --backend auto --max-runs 10 --sort-metric accuracy
```

### Find best model
```bash
python3 ${CLAUDE_SKILL_DIR}/../../scripts/compare_experiments.py best \
    --backend auto --metric accuracy --cost-weight 0.01
```

### Generate report
```bash
python3 ${CLAUDE_SKILL_DIR}/../../scripts/generate_report.py \
    --comparison-file comparison.json --recommendation-file recommendation.json
```

### Visualize training curves
```bash
python3 ${CLAUDE_SKILL_DIR}/../../scripts/visualize_curves.py demo --runs 3
```

## Workflow

When the user asks to compare experiments:

1. Detect backend: check for MLflow tracking server or SageMaker training jobs
2. Pull runs with metrics, params, cost data
3. Present comparison as Markdown table
4. Recommend best model with reasoning (accuracy vs cost vs time)
5. Offer to register winner to Model Registry

Always show results as formatted Markdown tables with clear recommendations.