lawwu Check Training
Monitor a machine learning training run and maintain session control until it reaches a terminal state.
Invocation
Arguments ($ARGUMENTS) are interpreted as:
- Local log file: a path like
./logs/train.logor/tmp/run.log - Remote log file:
user@host:/path/to/train.log(uses SSH) - Process by PID:
--pid 12345 - Process by name:
--proc torchrun,--proc train.py, or--proc "uv run" - Vertex AI Pipeline job:
--vertex-pipeline JOB_ID_OR_NAME(KFP/Vertex Pipelines) - Vertex AI Custom training job:
--vertex-training JOB_ID_OR_NAME - Wandb/MLflow run:
--wandb run-idor--mlflow run-id
Optional flags:
--project PROJECT— GCP project ID (required for Vertex AI targets; falls back togcloud config get-value project)--region REGION— GCP region (default:us-central1)--interval N— initial poll interval in seconds (default: 30; default 60 for Vertex AI)--max-interval N— cap for exponential backoff in seconds (default: 300)--no-backoff— disable exponential backoff; poll at fixed--interval--once— take a single snapshot and exit--no-fix— observe only, never attempt fixes
Target: $ARGUMENTS
Your responsibilities
1. Parse the target
Determine whether the target is local, remote, or cloud, and what monitoring strategy to use:
- Log file (local):
tail -n 100 <path>to read recent lines - Log file (remote):
ssh user@host 'tail -n 100 <path>'to read recent lines - PID:
ps -p <pid>to check if process is alive; read/proc/<pid>/fdor companion log if available - Process name:
pgrep -a <name>to find PIDs, then follow above - Vertex AI Pipeline: use
gcloudCLI or Python SDK — see references/vertex-ai.md - Vertex AI Custom training: use
gcloud ai custom-jobs describe— see references/vertex-ai.md - No argument given: ask the user for the target before proceeding
2. Establish a baseline
On first poll:
- Confirm the process is running and the log is readable
- Extract the current epoch/step/iteration number
- Note the current loss and any other reported metrics (accuracy, perplexity, grad norm, lr, etc.)
- Identify the log format (structured JSON, key=value pairs, plain text) — see references/log-formats.md
- Report a one-line status:
[Step X/Y] loss=Z, lr=A, gpu_util=B%
3. Continuous monitoring loop
Maintain a current interval starting at --interval. Unless --no-backoff is set, apply exponential backoff:
- After each uneventful poll (no anomaly, loss delta ≤5%, no new epoch/stage, no Vertex AI state change): multiply current interval by 1.5, capped at
--max-interval - Reset to
--intervalwhenever any of the following occurs:- An anomaly is detected (any severity)
- Loss delta > 5% since the previous poll
- A new epoch or training stage begins
- Vertex AI pipeline task state or job state changes
Poll on each interval. For each poll:
- Read the latest log lines (or query the metrics API)
- Extract the latest step/epoch and metrics
- Run anomaly checks (see references/common-issues.md)
- If anomalies are found, diagnose and decide whether to fix or escalate; reset interval to
--interval - Report a one-line progress update (include
next poll in Xs— see Section 6) - Update the current interval per backoff rules above
- Check for terminal conditions (see Stop Conditions below)
After any fix that pushes a change or restarts a process, restart the monitoring loop in the same turn — do not wait for the user to re-invoke.
4. Anomaly detection
For local/remote training runs, check on every poll:
| Issue | Signal | Severity |
|---|---|---|
| NaN/Inf loss | loss=nan or loss=inf in logs |
Critical |
| Loss divergence | Loss increases monotonically over 5+ steps | High |
| Stalled training | No new log lines for 2+ intervals | High |
| OOM / CUDA error | CUDA out of memory or RuntimeError in logs |
Critical |
| Gradient explosion | Grad norm > 1000 (or >> baseline) | High |
| Process crash | PID no longer alive | Critical |
| Checkpoint save failure | Error saving checkpoint |
Medium |
| Slow throughput | Samples/sec drops > 50% from baseline | Medium |
For Vertex AI Pipeline jobs, check on every poll:
| Issue | Signal | Severity |
|---|---|---|
| Pipeline failure | state: PIPELINE_STATE_FAILED |
Critical |
| Task failure | Any task in TASK_STATE_FAILED state |
High |
| Pipeline cancellation | state: PIPELINE_STATE_CANCELLED |
High |
| Stuck pipeline | State remains PIPELINE_STATE_RUNNING with no task state changes for 3+ intervals |
High |
| Error in task logs | Error-level messages in pipeline_job_task_events log |
Medium |
For Vertex AI Custom training jobs, check on every poll:
| Issue | Signal | Severity |
|---|---|---|
| Job failure | state: JOB_STATE_FAILED |
Critical |
| Job cancellation | state: JOB_STATE_CANCELLED |
High |
| Stalled job | State remains JOB_STATE_RUNNING with no log output for 3+ intervals |
High |
| OOM / CUDA error | Errors in Cloud Logging for the job | Critical |
For local/remote anomaly details and fix strategies, see references/common-issues.md. For Vertex AI-specific monitoring, see references/vertex-ai.md.
5. Fix policy
Apply fixes only when:
- The fix is clearly safe and reversible
--no-fixflag is NOT set- You have not already retried this type of fix 3 times
Safe fixes you may attempt:
- Restart a crashed process (if restart command is determinable from the log header or working directory)
- Clear CUDA cache (
torch.cuda.empty_cache()via a helper script if accessible) - Kill and restart a zombie process
Fixes that require user approval:
- Changing hyperparameters (learning rate, batch size)
- Rolling back to a checkpoint
- Canceling the run
Always state what you are about to do before doing it.
6. Progress reporting format
Local/remote training:
[HH:MM:SS] Step 1200/10000 (12%) | loss=0.342 | lr=1e-4 | grad_norm=0.8 | 142 samples/s
GPU: 94% util, 18.2/24 GB VRAM | ETA: ~1h 23m | next poll in 45s
If remote SSH, prefix with [user@host].
Vertex AI Pipeline job:
[HH:MM:SS] Pipeline: my-training-pipeline (projects/my-proj/locations/us-central1/pipelineJobs/123)
State: RUNNING | Elapsed: 1h 23m
Tasks: 3 SUCCEEDED, 1 RUNNING (data-preprocessing), 2 PENDING
Running task: model-training — started 45m ago
Vertex AI Custom training job:
[HH:MM:SS] Custom job: my-gpt2-finetune (projects/my-proj/locations/us-central1/customJobs/456)
State: RUNNING | Elapsed: 2h 10m | Worker: n1-standard-8 + 4x T4
Recent logs: Step 4200/10000 | loss=0.842
Report a full summary every 5 polls or when something notable happens.
7. Stop conditions
Stop monitoring and hand back control to the user when:
- Training complete: log indicates run finished (
Training finished,Epoch N/N done, process exits cleanly) - Vertex AI terminal state: pipeline or job reaches
SUCCEEDED,FAILED, orCANCELLED - Critical unrecoverable error: OOM with no restart option, NaN loss with no checkpoint to roll back to, 3 retries exhausted
- User intervention needed: ambiguous failure, hyperparameter decision needed, infrastructure issue outside Claude's reach
--onceflag was passed: after the first snapshot
When stopping, always output a final summary: total steps completed (or pipeline tasks), final metrics or error message, reason for stopping, and any recommended next steps.
Final summary format:
Training Complete
=================
Target: <log/job>
Terminal state: SUCCEEDED | FAILED | CANCELLED | TIMEOUT
Steps completed: <N>/<total>
Final metrics: <metric>=<value>, ...
Best checkpoint: <path or none>
Decision: GO | NO-GO
Confidence: high|medium|low
GO: training reached a terminal SUCCEEDED state with no unresolved critical anomalies.
NO-GO: terminal state is FAILED, CANCELLED, or TIMEOUT with unresolved blockers.
JSON artifact
Write babysit-training.json to --out-dir (or ./ if invoked standalone) following the schema in ../../references/schemas.md. Use vocabulary from ../../references/vocabulary.md.
Key fields to populate:
decision:GOwhen training succeeded;NO-GOwhen terminal state is a failureterminal_state,total_steps,final_metrics,best_checkpointanomalies_detected: one entry per anomaly observed during monitoringfindings: surface each critical/high anomaly as a finding
Example sessions
Local/remote training (with exponential backoff):
/ml-skills:babysit-training user@gpu-box:~/runs/gpt2-finetune/train.log --interval 30
[10:03:00] Connecting to gpu-box...
[10:03:01] Process found: PID 48291 (torchrun, 4 GPUs)
[10:03:01] Step 340/5000 (6.8%) | loss=2.41 | lr=3e-4 | grad_norm=1.2 | 98 samples/s
GPU: 97% util, 62/80 GB VRAM | ETA: ~13h 20m | next poll in 30s
[10:03:31] Step 342/5000 (6.8%) | loss=2.39 | lr=3e-4 | grad_norm=1.1 | 99 samples/s
GPU: 97% util | ETA: ~13h 18m | next poll in 45s ← backed off 30s→45s
[10:04:16] Step 345/5000 (6.9%) | loss=2.38 | lr=3e-4 | grad_norm=1.1 | 99 samples/s
GPU: 97% util | ETA: ~13h 15m | next poll in 68s ← backed off 45s→68s
[10:05:24] Step 349/5000 (7.0%) | loss=nan — ANOMALY: NaN loss detected
grad_norm spiked to 1842 at step 348. Last checkpoint: step 340 (2m ago).
Interval reset to 30s.
Recommended: roll back to step 340 and reduce lr. Awaiting your approval.
Vertex AI Pipeline:
/ml-skills:babysit-training --vertex-pipeline my-llm-pipeline --project my-gcp-proj --interval 120
[10:00:00] Pipeline: my-llm-pipeline
Job: projects/my-gcp-proj/locations/us-central1/pipelineJobs/1234567890
State: RUNNING | Started: 10 minutes ago
Tasks: 1 SUCCEEDED (data-validation), 1 RUNNING (model-training), 3 PENDING
[10:02:00] Tasks: 1 SUCCEEDED, 1 RUNNING (model-training — 12 min elapsed), 3 PENDING — OK
[10:04:00] Tasks: 1 SUCCEEDED, 1 FAILED (model-training), 3 PENDING — ANOMALY: Task failed
Fetching task logs from Cloud Logging...
Error: CUDA out of memory on worker 0. Allocated 23.8 GB, needed 24.5 GB.
Recommendation: reduce batch size or enable gradient checkpointing. Awaiting your approval.
Vertex AI Custom training job:
/ml-skills:babysit-training --vertex-training 9876543210 --project my-gcp-proj
[10:00:00] Custom job: projects/my-gcp-proj/locations/us-central1/customJobs/9876543210
State: RUNNING | Machine: n1-standard-8 + 2x NVIDIA_TESLA_V100
Recent log: Step 200/5000 | loss=2.31 | lr=3e-4
[10:00:30] State: SUCCEEDED | Duration: 4h 12m
Final log: Training complete. Best checkpoint: gs://my-bucket/checkpoints/step-4800/
Done. Monitoring complete.
Additional resources
- references/common-issues.md — Anomaly descriptions and fix strategies
- references/log-formats.md — How to parse common training log formats
- references/vertex-ai.md — Vertex AI Pipelines and Custom Jobs monitoring
- scripts/tail-remote.sh — Helper for tailing remote logs over SSH
- scripts/check-process.sh — Process health checker (local or remote)