Ml Experiment

ML Experiment — Full ML Pipeline

An agent team collaborates to perform the full ML experiment lifecycle: data preparation → model design → training → evaluation → deployment readiness.

Execution Mode

Agent Team — 5 members communicate directly via SendMessage and cross-validate.

Agent Composition

Workflow

Phase 1: Preparation (Orchestrator performs directly)

1. Extract from user input:
   • Problem Definition: Classification/regression/generation/recommendation/time-series, etc.
   • Data: Data source, files, format, scale
   • Target Metric: Specific goals such as accuracy, F1, RMSE
   • Constraints (optional): Framework, GPU, inference speed, model size
   • Existing Code (optional): Existing models, preprocessing code, experiment results
2. Create _workspace/ directory at the project root
3. Organize input and save to _workspace/00_input.md
4. If existing files are present, copy to _workspace/ and skip the corresponding Phase
5. Determine execution mode based on request scope

Phase 2: Team Assembly and Execution

Inter-team communication flow:

• data-engineer completes → Sends feature/shape/data characteristics to model-designer, data loader to training, class distribution to evaluation
• model-designer completes → Sends model code/hyperparameter space to training, model structure/evaluation metrics to evaluation
• training completes → Sends training curves/best model/experiment logs to evaluation
• evaluation completes → Sends evaluation report to reviewer
• reviewer cross-validates all outputs. If 🔴 must-fix issues found, sends correction requests to the relevant agent → rework → re-verify (up to 2 times)

Phase 3: Integration and Final Outputs

1. Check all files in _workspace/
2. Verify that all 🔴 must-fix items from the review report have been addressed
3. Report final summary to the user:
   • Data Preparation — 01_data_preparation.md
   • Model Design — 02_model_design.md
   • Training Configuration — 03_training_config.md
   • Evaluation Report — 04_evaluation_report.md
   • Review Report — 05_review_report.md
   • Experiment Code — experiment_code/

Scale-Based Modes

Leveraging existing files: If the user provides preprocessing code, trained models, etc., skip the corresponding steps.

Data Transfer Protocol

File naming convention: {order}_{agent}_{output}.{extension}

Error Handling

Test Scenarios

Normal Flow

Prompt: "Build a survival prediction classification model using the Kaggle Titanic dataset. Target F1 score above 0.85." Expected Results:

• Data: EDA (missing values, distributions, correlations), preprocessing pipeline (Imputer+Scaler+Encoder), stratified split
• Model: Baseline (LogisticRegression) + XGBoost + RandomForest design
• Training: Optuna hyperparameter tuning, MLflow experiment tracking
• Evaluation: Confusion Matrix, SHAP analysis, model comparison, statistical verification
• Review: No data leakage confirmed, reproducibility confirmed, conclusion validity verified

Existing File Flow

Prompt: "Evaluate this trained model and suggest improvement directions" + model file attached Expected Results:

• Copy existing model to _workspace/
• Evaluation mode: evaluation-analyst + reviewer deployed
• Performance analysis, error analysis, improvement recommendations provided

Error Flow

Prompt: "Build a machine learning model, but I don't have data yet" Expected Results:

• Request problem type confirmation
• Recommend 3-5 public datasets (UCI/Kaggle/HuggingFace)
• Provide synthetic data generation code
• State "Full pipeline can be executed after data acquisition"

Agent Extension Skills