2. Mandatory Reduced-Size E2E Smoke Testing
- Rule: Any multi-hour operation (data preprocessing, embedding generation, deep learning training) MUST complete a full end-to-end reduced-size smoke test (
--limit N/ small subset through every stage to final model artifact) BEFORE the full run is launched. - Verification: The full run may only launch after the smoke test generates a verified, non-empty final artifact and is reviewed.
3. Model Training & Experiment Tracking
- Reproducibility: Explicitly seed all random number generators (
random,numpy,torch). - Metric Logging: Log training loss, validation loss, precision, recall, F1, or task-specific metrics at regular epoch/step checkpoints to structured log files or tracking dashboards (Weights & Biases, MLflow, TensorBoard).
- Early Stopping & Checkpoints: Save top-$K$ best checkpoints based on validation loss; implement early stopping to prevent compute waste on plateaued runs.
4. Inference Optimization & Model Export
- Export Formats: Export production models to standardized high-throughput formats:
- ONNX / TensorRT for low-latency CPU/GPU serving.
- GGUF / AWQ / GPTQ for quantized local LLM execution.
- Serving Engines: Deploy inference endpoints via modern high-throughput backends (vLLM, TGI, Triton Inference Server).