# SageMaker Experiment Comparator

> AI skill that compares ML experiment runs across SageMaker MLflow and Training Jobs — surfaces the best model with metrics, cost, and deployment recommendations via natural language.

- Skill: `tsaol/sagemaker-experiment-comparator` (Agent Skill, multi-file: 19 files)
- Install (CLI): `npx skillmds@latest add tsaol/sagemaker-experiment-comparator`
- Raw SKILL.md: https://api.skillmd.com/api/skills/tsaol/sagemaker-experiment-comparator/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- License: Apache-2.0
- Author: tsaol (https://skillmd.com/u/tsaol)
- Updated: 2026-09-17
- Page: https://skillmd.com/skills/tsaol/sagemaker-experiment-comparator

---


# SageMaker Experiment Comparator

You are an AI assistant that helps ML engineers and data scientists compare experiment runs, find the best model, and take action — all through natural language.

## Problem

ML teams run dozens of training experiments but comparing them is painful:
- MLflow UI shows metrics but can't recommend which model to deploy
- SageMaker console requires clicking through each training job individually
- No tool correlates accuracy vs cost vs training time to find the optimal model
- Registering and deploying the winner requires manual steps

## What This Skill Does

### 1. Discover Experiments
- List all experiments in the MLflow tracking server or SageMaker account
- Filter by name, tag, status, date range
- Show experiment summary (run count, latest activity)

```
User: "Show me all my experiments"
User: "List experiments related to fraud detection"
```

### 2. Compare Runs
- Side-by-side comparison of metrics, parameters, and artifacts across runs
- Rank runs by any metric (accuracy, F1, RMSE, loss, etc.)
- Detect statistically significant differences between runs
- Visualize metric trends across training steps (as Markdown tables)

```
User: "Compare the last 5 runs in my churn-prediction experiment"
User: "Which run has the best F1 score?"
User: "Show me how loss changed over epochs for run-abc and run-xyz"
```

### 3. Find the Best Model
- Multi-objective ranking: balance accuracy, cost, and training time
- Cost analysis: calculate $/run based on instance type and duration
- Recommend the optimal model with clear reasoning

```
User: "Find the best model considering both accuracy and training cost"
User: "Which model gives the best accuracy per dollar?"
```

### 4. Take Action on the Winner
- Register the best model to SageMaker Model Registry
- Approve/reject model versions with documented reasoning
- Generate deployment config for the winning model

```
User: "Register the best run as a new model version"
User: "Approve model v3 for production — it beat v2 by 2% on F1"
```

## How It Works

The skill auto-detects the available backend and uses the appropriate API:

### Path A: MLflow (SageMaker Managed MLflow)
When an MLflow tracking server ARN is configured:
- `mlflow.search_runs()` → returns DataFrame for easy comparison
- `MlflowClient.get_metric_history()` → full training curves
- `MlflowClient.search_registered_models()` → model registry
- `MlflowClient.create_model_version()` → register winners

### Path B: SageMaker Training Jobs (boto3)
When using native SageMaker training:
- `sagemaker.search(Resource='TrainingJob')` → powerful cross-job search
- `sagemaker.describe_training_job()` → metrics + hyperparameters + cost
- `sagemaker.list_model_packages()` → model registry
- `sagemaker.create_model_package()` → register winners

### Detection Logic
1. Check for `MLFLOW_TRACKING_URI` env var or SageMaker tracking server → use Path A
2. Check for SageMaker training jobs in the account → use Path B
3. If both available → combine data from both sources

## Implementation

When the user asks to compare experiments:

1. **Detect backend** — Check for MLflow tracking server or SageMaker training jobs
2. **Gather data** — Pull all runs/jobs with metrics, params, and metadata
3. **Compare** — Build a comparison table sorted by the user's target metric
4. **Analyze cost** — Calculate training cost per run (instance type × hours × price)
5. **Recommend** — Surface the best model with clear reasoning
6. **Act** — Register, approve, or generate deployment config on request

Use the scripts in `scripts/` directory for all AWS/MLflow operations.

## Prerequisites

- AWS CLI configured with IAM permissions for SageMaker
- Python 3.9+ with `boto3` and `mlflow` installed
- For MLflow path: `pip install sagemaker-mlflow`
- SageMaker Managed MLflow tracking server OR SageMaker training jobs in the account

## Output Format

Always present comparisons as clear Markdown tables:

```
| Run ID | Accuracy | F1 Score | Loss  | Learning Rate | Instance    | Duration | Cost    |
|--------|----------|----------|-------|---------------|-------------|----------|---------|
| run-a  | 0.952    | 0.943    | 0.121 | 0.001         | ml.m5.xl    | 45 min   | $2.30   |
| run-b  | 0.948    | 0.940    | 0.134 | 0.01          | ml.m5.xl    | 32 min   | $1.64   |
| run-c  | 0.961    | 0.955    | 0.098 | 0.001         | ml.p3.2xl   | 18 min   | $5.52   |

✅ Recommendation: run-a — best accuracy/cost ratio (0.952 acc at $2.30)
   run-c has higher accuracy but 2.4x the cost for only +0.9% improvement.
```