Validation Scripts

ML Training Validation Scripts

Purpose: Production-ready validation and testing utilities for ML training workflows. Ensures data quality, model integrity, pipeline correctness, and dependency availability before and during training.

Activation Triggers:

• Validating training datasets before fine-tuning
• Checking model checkpoints and outputs
• Testing end-to-end training pipelines
• Verifying system dependencies and GPU availability
• Debugging training failures or data issues
• Ensuring data format compliance
• Validating model configurations
• Testing inference pipelines

Key Resources:

• scripts/validate-data.sh - Comprehensive dataset validation
• scripts/validate-model.sh - Model checkpoint and config validation
• scripts/test-pipeline.sh - End-to-end pipeline testing
• scripts/check-dependencies.sh - System and package dependency checking
• templates/test-config.yaml - Pipeline test configuration template
• templates/validation-schema.json - Data validation schema template
• examples/data-validation-example.md - Dataset validation workflow
• examples/pipeline-testing-example.md - Complete pipeline testing guide

Quick Start

Validate Training Data

bash scripts/validate-data.sh \
  --data-path ./data/train.jsonl \
  --format jsonl \
  --schema templates/validation-schema.json

Validate Model Checkpoint

bash scripts/validate-model.sh \
  --model-path ./checkpoints/epoch-3 \
  --framework pytorch \
  --check-weights

Test Complete Pipeline

bash scripts/test-pipeline.sh \
  --config templates/test-config.yaml \
  --data ./data/sample.jsonl \
  --verbose

Check System Dependencies

bash scripts/check-dependencies.sh \
  --framework pytorch \
  --gpu-required \
  --min-vram 16

Validation Scripts

1. Data Validation (validate-data.sh)

Purpose: Validate training datasets for format compliance, data quality, and schema conformance.

Usage:

bash scripts/validate-data.sh [OPTIONS]

Options:

• --data-path PATH - Path to dataset file or directory (required)
• --format FORMAT - Data format: jsonl, csv, parquet, arrow (default: jsonl)
• --schema PATH - Path to validation schema JSON file
• --sample-size N - Number of samples to validate (default: all)
• --check-duplicates - Check for duplicate entries
• --check-null - Check for null/missing values
• --check-length - Validate text length constraints
• --check-tokens - Validate tokenization compatibility
• --tokenizer MODEL - Tokenizer to use for token validation (default: gpt2)
• --max-length N - Maximum sequence length (default: 2048)
• --output REPORT - Output validation report path (default: validation-report.json)

Validation Checks:

• Format Compliance: Verify file format and structure
• Schema Validation: Check against JSON schema if provided
• Data Types: Validate field types (string, int, float, etc.)
• Required Fields: Ensure all required fields are present
• Value Ranges: Check numeric values within expected ranges
• Text Quality: Detect empty strings, excessive whitespace, encoding issues
• Duplicates: Identify duplicate entries (exact or near-duplicate)
• Token Counts: Verify sequences fit within model context length
• Label Distribution: Check class balance for classification tasks
• Missing Values: Detect and report null/missing values

Example Output:

{
  "status": "PASS",
  "total_samples": 10000,
  "valid_samples": 9987,
  "invalid_samples": 13,
  "validation_errors": [
    {
      "sample_id": 42,
      "field": "text",
      "error": "Exceeds max token length: 2150 > 2048"
    },
    {
      "sample_id": 156,
      "field": "label",
      "error": "Invalid label value: 'unknwn' (typo)"
    }
  ],
  "statistics": {
    "avg_text_length": 487,
    "avg_token_count": 128,
    "label_distribution": {
      "positive": 4892,
      "negative": 4895,
      "neutral": 213
    }
  },
  "recommendations": [
    "Remove or fix 13 invalid samples before training",
    "Label distribution is imbalanced - consider class weighting"
  ]
}

Exit Codes:

• 0 - All validations passed
• 1 - Validation errors found
• 2 - Script error (invalid arguments, file not found)

2. Model Validation (validate-model.sh)

Purpose: Validate model checkpoints, configurations, and inference readiness.

Usage:

bash scripts/validate-model.sh [OPTIONS]

Options:

• --model-path PATH - Path to model checkpoint directory (required)
• --framework FRAMEWORK - Framework: pytorch, tensorflow, jax (default: pytorch)
• --check-weights - Verify model weights are loadable
• --check-config - Validate model configuration file
• --check-tokenizer - Verify tokenizer files are present
• --check-inference - Test inference with sample input
• --sample-input TEXT - Sample text for inference test
• --expected-output TEXT - Expected output for verification
• --output REPORT - Output validation report path

Validation Checks:

• File Structure: Verify required files are present
• Config Validation: Check model_config.json for correctness
• Weight Integrity: Load and verify model weights
• Tokenizer Files: Ensure tokenizer files exist and are loadable
• Model Architecture: Validate architecture matches config
• Memory Requirements: Estimate GPU/CPU memory needed
• Inference Test: Run sample inference if requested
• Quantization: Verify quantized models load correctly
• LoRA/PEFT: Validate adapter weights and configuration

Example Output:

{
  "status": "PASS",
  "model_path": "./checkpoints/llama-7b-finetuned",
  "framework": "pytorch",
  "checks": {
    "file_structure": "PASS",
    "config": "PASS",
    "weights": "PASS",
    "tokenizer": "PASS",
    "inference": "PASS"
  },
  "model_info": {
    "architecture": "LlamaForCausalLM",
    "parameters": "7.2B",
    "precision": "float16",
    "lora_enabled": true,
    "lora_rank": 16
  },
  "memory_estimate": {
    "model_size_gb": 13.5,
    "inference_vram_gb": 16.2,
    "training_vram_gb": 24.8
  },
  "inference_test": {
    "input": "Hello, world!",
    "output": "Hello, world! How can I help you today?",
    "latency_ms": 142
  }
}

3. Pipeline Testing (test-pipeline.sh)

Purpose: Test end-to-end ML training and inference pipelines with sample data.

Usage:

bash scripts/test-pipeline.sh [OPTIONS]

Options:

• --config PATH - Pipeline test configuration file (required)
• --data PATH - Sample data for testing
• --steps STEPS - Comma-separated steps to test (default: all)
• --verbose - Enable detailed output
• --output REPORT - Output test report path
• --fail-fast - Stop on first failure
• --cleanup - Clean up temporary files after test

Pipeline Steps:

• data_loading: Test data loading and preprocessing
• tokenization: Verify tokenization pipeline
• model_loading: Load model and verify initialization
• training_step: Run single training step
• validation_step: Run single validation step
• checkpoint_save: Test checkpoint saving
• checkpoint_load: Test checkpoint loading
• inference: Test inference pipeline
• metrics: Verify metrics calculation

Test Configuration (test-config.yaml):

pipeline:
  name: llama-7b-finetuning
  framework: pytorch

data:
  train_path: ./data/sample-train.jsonl
  val_path: ./data/sample-val.jsonl
  format: jsonl

model:
  base_model: meta-llama/Llama-2-7b-hf
  checkpoint_path: ./checkpoints/test
  load_in_8bit: false
  lora:
    enabled: true
    r: 16
    alpha: 32

training:
  batch_size: 1
  gradient_accumulation: 1
  learning_rate: 2e-4
  max_steps: 5

testing:
  sample_size: 10
  timeout_seconds: 300
  expected_loss_range: [0.5, 3.0]

Example Output:

Pipeline Test Report
====================

Pipeline: llama-7b-finetuning
Started: 2025-11-01 12:34:56
Duration: 127 seconds

Test Results:
✓ data_loading (2.3s) - Loaded 10 samples successfully
✓ tokenization (1.1s) - Tokenized all samples, avg length: 128 tokens
✓ model_loading (8.7s) - Model loaded, 7.2B parameters
✓ training_step (15.4s) - Training step completed, loss: 1.847
✓ validation_step (12.1s) - Validation step completed, loss: 1.923
✓ checkpoint_save (3.2s) - Checkpoint saved to ./checkpoints/test
✓ checkpoint_load (6.8s) - Checkpoint loaded successfully
✓ inference (2.9s) - Inference completed, latency: 142ms
✓ metrics (0.4s) - Metrics calculated correctly

Overall: PASS (8/8 tests passed)

Performance Metrics:
- Total time: 127s
- GPU memory used: 15.2 GB
- CPU memory used: 8.4 GB

Recommendations:
- Pipeline is ready for full training
- Consider increasing batch_size to improve throughput

4. Dependency Checking (check-dependencies.sh)

Purpose: Verify all required system dependencies, packages, and GPU availability.

Usage:

bash scripts/check-dependencies.sh [OPTIONS]

Options:

• --framework FRAMEWORK - ML framework: pytorch, tensorflow, jax (default: pytorch)
• --gpu-required - Require GPU availability
• --min-vram GB - Minimum GPU VRAM required (GB)
• --cuda-version VERSION - Required CUDA version
• --packages FILE - Path to requirements.txt or packages list
• --platform PLATFORM - Platform: modal, lambda, runpod, local (default: local)
• --fix - Attempt to install missing packages
• --output REPORT - Output dependency report path

Dependency Checks:

• Python Version: Verify compatible Python version
• ML Framework: Check PyTorch/TensorFlow/JAX installation
• CUDA/cuDNN: Verify CUDA toolkit and cuDNN
• GPU Availability: Detect and validate GPUs
• VRAM: Check GPU memory capacity
• System Packages: Verify system-level dependencies
• Python Packages: Check required pip packages
• Platform Tools: Validate platform-specific tools (Modal, Lambda CLI)
• Storage: Check available disk space
• Network: Test internet connectivity for model downloads

Example Output:

{
  "status": "PASS",
  "platform": "modal",
  "checks": {
    "python": {
      "status": "PASS",
      "version": "3.10.12",
      "required": ">=3.9"
    },
    "pytorch": {
      "status": "PASS",
      "version": "2.1.0",
      "cuda_available": true,
      "cuda_version": "12.1"
    },
    "gpu": {
      "status": "PASS",
      "count": 1,
      "type": "NVIDIA A100",
      "vram_gb": 40,
      "driver_version": "535.129.03"
    },
    "packages": {
      "status": "PASS",
      "installed": 42,
      "missing": 0,
      "outdated": 3
    },
    "storage": {
      "status": "PASS",
      "available_gb": 128,
      "required_gb": 50
    }
  },
  "recommendations": [
    "Update transformers to latest version (4.36.0 -> 4.37.2)",
    "Consider upgrading to PyTorch 2.2.0 for better performance"
  ]
}

Templates

Validation Schema Template (templates/validation-schema.json)

Defines expected data structure and validation rules for training datasets.

{
  "$schema": "http://json-schema.org/draft-07/schema#",
  "type": "object",
  "required": ["text", "label"],
  "properties": {
    "text": {
      "type": "string",
      "minLength": 10,
      "maxLength": 5000,
      "description": "Input text for training"
    },
    "label": {
      "type": "string",
      "enum": ["positive", "negative", "neutral"],
      "description": "Classification label"
    },
    "metadata": {
      "type": "object",
      "properties": {
        "source": {
          "type": "string"
        },
        "timestamp": {
          "type": "string",
          "format": "date-time"
        }
      }
    }
  },
  "validation_rules": {
    "max_token_length": 2048,
    "tokenizer": "meta-llama/Llama-2-7b-hf",
    "check_duplicates": true,
    "min_label_count": 100,
    "max_label_imbalance_ratio": 10.0
  }
}

Customization:

• Modify required fields for your use case
• Add custom properties and validation rules
• Set appropriate length constraints
• Define label enums for classification
• Configure tokenizer and max length

Test Configuration Template (templates/test-config.yaml)

Defines pipeline testing parameters and expected behaviors.

pipeline:
  name: test-pipeline
  framework: pytorch
  platform: modal  # modal, lambda, runpod, local

data:
  train_path: ./data/sample-train.jsonl
  val_path: ./data/sample-val.jsonl
  format: jsonl  # jsonl, csv, parquet
  sample_size: 10  # Number of samples to use for testing

model:
  base_model: meta-llama/Llama-2-7b-hf
  checkpoint_path: ./checkpoints/test
  quantization: null  # null, 8bit, 4bit
  lora:
    enabled: true
    r: 16
    alpha: 32
    dropout: 0.05
    target_modules: ["q_proj", "v_proj"]

training:
  batch_size: 1
  gradient_accumulation: 1
  learning_rate: 2e-4
  max_steps: 5
  warmup_steps: 0
  eval_steps: 2

testing:
  sample_size: 10
  timeout_seconds: 300
  expected_loss_range: [0.5, 3.0]
  fail_fast: true
  cleanup: true

validation:
  metrics: ["loss", "perplexity"]
  check_gradients: true
  check_memory_leak: true

gpu:
  required: true
  min_vram_gb: 16
  allow_cpu_fallback: false

Integration with ML Training Workflow

Pre-Training Validation

# 1. Check system dependencies
bash scripts/check-dependencies.sh \
  --framework pytorch \
  --gpu-required \
  --min-vram 16

# 2. Validate training data
bash scripts/validate-data.sh \
  --data-path ./data/train.jsonl \
  --schema templates/validation-schema.json \
  --check-duplicates \
  --check-tokens

# 3. Test pipeline with sample data
bash scripts/test-pipeline.sh \
  --config templates/test-config.yaml \
  --verbose

Post-Training Validation

# 1. Validate model checkpoint
bash scripts/validate-model.sh \
  --model-path ./checkpoints/final \
  --framework pytorch \
  --check-weights \
  --check-inference

# 2. Test inference pipeline
bash scripts/test-pipeline.sh \
  --config templates/test-config.yaml \
  --steps inference,metrics

CI/CD Integration

# .github/workflows/validate-training.yml
- name: Validate Training Data
  run: |
    bash plugins/ml-training/skills/validation-scripts/scripts/validate-data.sh \
      --data-path ./data/train.jsonl \
      --schema ./validation-schema.json

- name: Test Training Pipeline
  run: |
    bash plugins/ml-training/skills/validation-scripts/scripts/test-pipeline.sh \
      --config ./test-config.yaml \
      --fail-fast

Error Handling and Debugging

Common Validation Failures

Data Validation Failures:

• Token length exceeded → Truncate or filter long sequences
• Missing required fields → Fix data preprocessing
• Duplicate entries → Deduplicate dataset
• Invalid labels → Clean label data

Model Validation Failures:

• Missing config files → Ensure complete checkpoint
• Weight loading errors → Check framework compatibility
• Tokenizer mismatch → Verify tokenizer version

Pipeline Test Failures:

• Out of memory → Reduce batch size, enable gradient checkpointing
• CUDA errors → Check GPU drivers, CUDA version
• Slow performance → Profile and optimize data loading

Debug Mode

All scripts support verbose debugging:

bash scripts/validate-data.sh --data-path ./data/train.jsonl --verbose --debug

Outputs:

• Detailed progress logs
• Stack traces for errors
• Performance metrics
• Memory usage statistics

Best Practices

1. Always validate before training - Catch data issues early
2. Use schema validation - Enforce data quality standards
3. Test with sample data first - Verify pipeline before full training
4. Check dependencies on new platforms - Avoid runtime failures
5. Validate checkpoints - Ensure model integrity before deployment
6. Monitor validation metrics - Track data quality over time
7. Automate validation in CI/CD - Prevent bad data from entering pipeline
8. Keep validation schemas updated - Reflect current data requirements

Performance Tips

• Use --sample-size for quick validation of large datasets
• Run dependency checks once per environment, cache results
• Parallelize validation with GNU parallel for multi-file datasets
• Use --fail-fast in CI/CD to save time
• Cache tokenizers to avoid re-downloading

Troubleshooting

Script Not Found:

# Ensure you're in the correct directory
cd /path/to/ml-training/skills/validation-scripts
bash scripts/validate-data.sh --help

Permission Denied:

# Make scripts executable
chmod +x scripts/*.sh

Missing Dependencies:

# Install required tools
pip install jsonschema pandas pyarrow
sudo apt-get install jq bc

Plugin: ml-training Version: 1.0.0 Supported Frameworks: PyTorch, TensorFlow, JAX Platforms: Modal, Lambda Labs, RunPod, Local