Team Phai Ph Norges Gruppen Train GCP

GCP Training Pipeline

Train YOLOv8 models on a dedicated GCP VM (a2-highgpu-8g, 8x A100 40GB GPUs, 96 vCPU, 680 GB RAM). All training runs inside Docker containers via scripts/gcp-train.sh.

Prerequisites

• gcloud CLI authenticated with A100 GPU quota (training in asia-southeast1-c)
• Training data uploaded to gs://nmiai-norgesgruppen-data/
• No local GPU required — all training runs remotely

Quick Reference

First-Time Setup

scripts/gcp-train.sh create    # provisions VM, builds Docker image, syncs code
scripts/gcp-train.sh prepare   # converts COCO → YOLO on VM (in Docker)

This is idempotent — safe to re-run. If the VM already exists, it skips creation and just ensures the Docker image is built and code is synced.

Training Iteration Loop

# 1. Push code changes to VM
scripts/gcp-train.sh sync

# 2. Launch training (uses 4 L4s by default)
scripts/gcp-train.sh train --model m --epochs 100 --name exp01

# 3. Monitor
scripts/gcp-train.sh progress exp01    # metrics summary
scripts/gcp-train.sh logs exp01        # live log stream
scripts/gcp-train.sh tensorboard       # graphical monitoring (port 6006)

# 4. Download weights when done
scripts/gcp-train.sh pull exp01

# 5. Optionally zip & upload entire run to GCS for safekeeping
scripts/gcp-train.sh push-gcs exp01

# List/download archived runs from GCS
scripts/gcp-train.sh list-gcs
scripts/gcp-train.sh pull-gcs exp01_20260320-011500.zip

# 6. Package submission
./package.sh runs/train/exp01/weights/best.pt

Train arguments

Monitoring

• progress [name]: Parses results.csv, shows best mAP50, last 10 epochs formatted. Defaults to latest run if name omitted.
• status: Quick overview — container running/stopped, nvidia-smi for all GPUs, run listing.
• logs [name]: Docker container logs. Ctrl+C to detach.
• tensorboard: Launches TensorBoard in a separate container on port 6006.

VM Lifecycle

scripts/gcp-train.sh stop      # pause billing (disk charges still apply)
scripts/gcp-train.sh start     # resume VM
scripts/gcp-train.sh destroy   # delete training VM

Experiment Naming

Use descriptive names for tracking experiments:

scripts/gcp-train.sh train --model m --epochs 100 --name baseline-m-100ep
scripts/gcp-train.sh train --model l --epochs 150 --imgsz 1280 --name large-1280-150ep
scripts/gcp-train.sh train --model m --epochs 50 --name resume-exp --resume
scripts/gcp-train.sh train --model x --epochs 200 --device 0,1 --name xlarge-2gpu

Configuration

Override defaults via environment variables:

INSTANCE=nmiai-train      # VM name (default)
ZONE=asia-southeast1-c    # GCP zone (default)
PROJECT=my-project        # GCP project (default: gcloud config)

Troubleshooting

• GPU quota error: Need A100 GPUs. Current VMs in asia-southeast1-c / asia-southeast1-a.
• OOM during training: Reduce --batch (try 8 or -1 for auto) or --imgsz.
• NVIDIA driver issues: The setup script installs prebuilt kernel modules. If nvidia-smi fails, try rebooting: scripts/gcp-train.sh ssh -- sudo reboot
• Container exited: Check scripts/gcp-train.sh logs for error details.
• Code not updated: Always sync before train to push latest src/ + training/ changes.

Source: team-phai/ph-norges-gruppen — distributed by TomeVault.