Sagemaker Mlops

Amazon SageMaker MLOps

When to Use

• Designing ML training and serving infrastructure on AWS SageMaker
• Building ML pipelines with SageMaker Pipelines (training → evaluation → deployment)
• Implementing model versioning with SageMaker Model Registry
• Monitoring models in production with SageMaker Model Monitor
• Optimizing training costs with Spot instances and managed checkpointing
• Preparing for AWS Certified Machine Learning Engineer Associate (MLA-C01) exam

Core Jobs

1. Training Job Configuration

Spot training requirements:

• Must implement checkpointing (save model state periodically to S3)
• On interruption, SageMaker saves checkpoint; resumes from last checkpoint when capacity returns
• Checkpoint path: s3://bucket/checkpoints/job-name/
• Training frameworks (TensorFlow, PyTorch, MXNet) have native checkpoint support

Managed Spot Training code:

estimator = Estimator(
    ...
    use_spot_instances=True,
    max_run=3600,          # max total training time (seconds)
    max_wait=7200,         # max wait including interruptions
    checkpoint_s3_uri="s3://bucket/checkpoints/",
    checkpoint_local_path="/opt/ml/checkpoints"
)

Built-in algorithms vs custom containers:

2. Inference Endpoint Types

Async endpoint: request queued in SQS; processing result written to S3; notification via SNS/EventBridge.

Batch Transform: no endpoint needed; input from S3; output to S3; best for periodic bulk scoring.

Multi-model endpoint (MME): host thousands of models on a single endpoint; SageMaker loads/unloads models from S3 to GPU/CPU memory dynamically. Cost-effective for many similar models.

Multi-container endpoint: run different models/containers on one endpoint; invoke a specific container. Use for A/B testing or ensemble inference.

3. SageMaker Pipelines (MLOps Workflow)

Supported step types:

Example pipeline flow:

ProcessingStep (feature engineering)
    ↓
TrainingStep (train XGBoost)
    ↓
ProcessingStep (evaluate on test set)
    ↓
ConditionStep (accuracy > 0.9?)
    ├── Yes → RegisterModel (Approved)
    └── No → RegisterModel (Rejected)

SageMaker Pipelines vs Step Functions:

• Pipelines = ML-native; step types understand ML artifacts (models, datasets); experiment tracking built-in
• Step Functions = general workflow; use when integrating ML with non-ML AWS services

4. Model Registry

• Version ML models with metadata, metrics, and approval status
• Approval states: Pending (default) → Approved / Rejected
• CI/CD trigger: approved model version → EventBridge → CodePipeline → deploy to endpoint
• Cross-account: share model package groups across accounts (for separate dev/staging/prod accounts)

Workflow:

1. Training pipeline registers new model version (status: Pending)
2. Automatic evaluation → conditional approval or human review
3. Approval → EventBridge event → CodePipeline deploys to staging endpoint
4. Staging validation passes → deploy to production endpoint

5. Model Monitor

Continuously monitors deployed endpoint data for:

Setup requirements:

1. Enable data capture on endpoint (captures request/response samples to S3)
2. Generate baseline statistics from training data
3. Schedule monitoring job (hourly, daily, etc.)
4. CloudWatch alerts on constraint violations → SNS notification

6. SageMaker Feature Store

Feature reuse: compute features once, store in Feature Store, reuse across multiple models and teams. Point-in-time queries: offline store supports time-travel queries (get feature values as of specific timestamp) — prevents training/serving skew.

Key Concepts

• SageMaker Studio — unified web IDE: notebooks, experiments, pipelines, model registry, endpoints; replaces individual SageMaker interfaces
• SageMaker Experiments — track training runs, hyperparameters, metrics, artifacts; query to find best run
• SageMaker Clarify — bias detection and explainability (SHAP values) for training data and predictions
• SageMaker Debugger — capture tensors during training; detect training issues (vanishing gradients, overfitting)
• Hyperparameter Tuning (HPO) — Bayesian optimization or random search over defined hyperparameter ranges
• Model Dashboard — unified view of all models, endpoint health, monitor violations
• Inference Recommender — benchmark instance types for your model (right-sizing for cost/latency)
• SageMaker JumpStart — pre-trained models and solution templates (foundation models, computer vision, NLP)

Checklist

• Spot training enabled with checkpointing for long training jobs?
• Endpoint type matched to use case (real-time, async, batch, serverless)?
• SageMaker Pipelines defined for reproducible ML workflow (not ad-hoc notebooks)?
• Model Registry used for versioning and approval workflow?
• Data capture enabled on endpoint before setting up Model Monitor?
• Model Monitor baseline generated from training data statistics?
• Feature Store used for shared features across models (avoid feature duplication)?
• IAM execution roles for training jobs follow least-privilege principle?

Output Format

• 🔴 Critical — Spot training without checkpointing (job restarts from scratch on interruption, wasting compute); no model versioning (cannot roll back; no approval workflow)
• 🟡 Warning — Real-time endpoint for large payload inference (use async); no Model Monitor (production drift undetected); all training on on-demand instances (significant cost savings missed)
• 🟢 Suggestion — Multi-model endpoint for many similar models (cost savings vs individual endpoints); SageMaker Inference Recommender for instance right-sizing; Feature Store for cross-team feature reuse

Exam Tips

• Spot training = up to 90% cost savings; must enable checkpointing for long jobs — SageMaker resumes from last checkpoint after interruption
• Async endpoint = for payloads > 6MB or processing > 60 seconds; results written to S3; poll or use SNS for completion notification
• Batch Transform = offline scoring of entire dataset; no endpoint required; input from S3, output to S3
• Model Monitor requires baseline from training data statistics; monitors for data drift in production (compare incoming request distributions)
• SageMaker Pipelines = NOT Step Functions; native ML pipeline service with ML-specific steps (ProcessingStep, TrainingStep, ConditionStep, RegisterModel)
• Feature Store: online (low-latency serving) + offline (S3-backed, for training) — same concept as Vertex AI Feature Store; online and offline stores are separate
• ConditionStep = branch pipeline based on evaluation metric threshold (if accuracy > 0.9 → approve model; else → reject)
• Multi-model endpoint (MME) = host thousands of models on one endpoint; SageMaker dynamically loads/evicts models from memory based on traffic