Build Baseline

Build Baseline

Construct and evaluate non-ML baselines to establish the performance floor that any ML model must beat before being considered worthwhile.

Invocation

Arguments ($ARGUMENTS) are interpreted as:

• <objective> — plain-language description of the prediction goal
• --data SOURCE — dataset path, HuggingFace dataset, or DB target
• --label-col COL — target column name
• --metric METRIC — primary evaluation metric (e.g. f1, auprc, rmse, ndcg)
• --split — evaluation split (default: validation; use test only if no validation split exists)
• --out-dir DIR — output directory (default: ./)
• --run-id ID — run ID to attach to artifact (set by orchestrate-e2e; else generate locally)

Target: $ARGUMENTS

Your responsibilities

1. Load the dataset and identify the task type

Load the data from --data and inspect:

• Label column distribution (--label-col)
• Column dtypes and sample values
• Row counts per split

Classify the task as one of:

• classification — discrete label with finite cardinality
• regression — continuous numeric target
• ranking — labels express preference or relevance order
• forecasting — time-ordered target with temporal structure

If task type is ambiguous, state your assumption and explain it.

2. Build non-ML baselines appropriate to the task type

Use uv run python for all computation. Do not use pre-trained ML models — baselines must be fully non-ML.

Classification

Regression

Ranking

Forecasting

Build all methods applicable to the detected task type. Skip methods that cannot be computed from the available data (e.g., seasonal-naive when no seasonal period is detectable) and record the reason.

3. Evaluate each baseline on the primary metric

Evaluate on the --split set (default: validation) using the metric specified in --metric.

Supported metrics (use scikit-learn or equivalent):

• Classification: accuracy, f1 (macro), f1_binary, auprc, roc_auc, precision, recall
• Regression: rmse, mae, mape, r2
• Ranking: ndcg, map, mrr
• Forecasting: rmse, mae, smape

Show the evaluation command before running it.

4. Select the best baseline

• For metrics where higher is better (accuracy, f1, auprc, roc_auc, ndcg, map, r2): select the baseline with the highest score.
• For metrics where lower is better (rmse, mae, mape): select the baseline with the lowest score.
• Mark the winning method as is_best: true in the artifact.

The best baseline score is the floor that any ML model must exceed to justify training complexity.

5. Report format

Baseline Report
===============
Objective: <...>
Data: <...>
Label col: <...>
Task type: <classification|regression|ranking|forecasting>
Metric: <...>
Split: <validation|test>

Baselines evaluated:
| Method                  | Score    | Notes                          |
|-------------------------|----------|--------------------------------|
| majority-class          | 0.612    | predicts class 0 always        |
| stratified-random       | 0.489    |                                |
| single-feature-threshold| 0.731  ★ | age > 45 → churn=1 (best)      |
| zero-r                  | 0.612    |                                |

Best baseline: single-feature-threshold (score: 0.731)
Score to beat: 0.731 f1

Decision: GO

The ★ marks the winning method. Always report the score to beat in the final line.

6. Fix policy

Apply without approval:

• Loading and reading data (read-only)
• Running baseline computations on the validation split
• Writing the JSON artifact and any summary outputs

Require user approval before:

• Evaluating on the test split
• Modifying or writing back to the source data

7. Stop conditions

Stop when:

• A complete baseline report is produced and build-baseline.json is written, or
• The dataset cannot be loaded (report the specific error and stop), or
• No baseline method can be computed (report why and stop)

Quick heuristics

• Majority-class score suspiciously high (> 0.9) → check for severe class imbalance; flag as high finding
• Single-feature threshold beats all other baselines by large margin → that feature may be a leakage proxy; flag for review-target re-check
• Baseline outperforms a subsequently trained ML model → ML model likely has a bug, data issue, or the problem does not warrant ML (Rule 1 of Google's Rules of ML)
• No validation split available → fall back to 80/20 train/validation split from training data; document assumption

JSON artifact

Write build-baseline.json to --out-dir (or ./ if invoked standalone) following the base schema in ../../references/schemas.md. Use vocabulary from ../../references/vocabulary.md.

decision is always GO — baselines are informational and never block the pipeline.

Skill-specific fields to populate (in addition to base schema fields):

{
  "task_type": "classification | regression | ranking | forecasting",
  "data_source": "<path or dataset>",
  "label_col": "<col>",
  "primary_metric": "<metric>",
  "baselines": [
    {
      "method": "<name>",
      "description": "<what it does>",
      "score": 0.731,
      "is_best": true
    }
  ],
  "best_baseline": "<method name>",
  "best_score": 0.731
}

See ../../references/schemas.md for the full schema definition.