Dataset Evaluation

Workflow Instruction

Follow the workflow shown below. Locate the dataset, check the file type, and resolve any issues with missing files or wrong file types. Determine the fine-tuning model and fine-tuning strategy. Run scripts/format_detector.py to evaluate whether the file is formatted correctly for the currently selected model and strategy. Summarize the results: is the dataset ready for fine-tuning?

Workflow

1. Locate Dataset:

   • The full path may be a local file path, or an S3 URI
   • Resolve the full path to the dataset file, make sure read permissions are available, and help the user if the file is not found

2. Determine strategy and model:

   • File formatting depends on the currently selected fine-tuning strategy and fine-tuning base model.
   • If the strategy and model are already known from the conversation context (e.g., selected via the finetuning-setup skill), use them.
   • If not available in context, activate the finetuning-setup skill to determine them before proceeding.

3. Check File Formatting: Run the tool format_detector.py to make sure the file conforms to formatting requirements.

   • Send the full path directly to the format_detector script as an argument
   • Do not send the model and strategy as arguments
   • Do not download data from S3
   • Do not make local copies of data

4. Summarize Results: Tell the user if their data is ready

   • Examine the output of format_detector and compare to the known strategy and model
   • Important: training datasets and evaluation datasets have different format requirements.
     • Training datasets must match the fine-tuning strategy format (SFT, DPO, RLVR) per references/strategy_data_requirements.md
     • Evaluation datasets (for model evaluation) must match one of the SageMaker evaluation dataset formats.
   • Report back to the user if their current dataset is valid for its intended purpose
   • Warn the user if their dataset is valid, but for a different strategy or model
   • Warn the user if their dataset is not valid for any strategy/model pair

Messages to the User

• Introduction: "This skill checks the structure of your dataset for model fine-tuning."
• File types: This skill applies to files that are formatted according to the Amazon SageMaker AI Developer Guide

Resources

• scripts/format_detector.py is self-contained format validation script that can be run independently
• finetuning-setup skill should have already determined the fine-tuning strategy and base model
• references/strategy_data_requirements.md contains data format requirements per strategy

Script Details

• scripts/format_detector.py is self-contained format validation script that can be run independently:

# With the file path argument identified in workflow step 1
python scripts/format_detector.py local_path/to/dataset