Nexar Collision Eval

nexar-collision-eval

Auto Researching, not hyperparameter tuning: Convergence Analysis of 10,000 Experiments — Li (2026) (arXiv:2603.15916, 2026)

What this evaluates

Evaluates autonomous ML research agents on their ability to search a mixed categorical-continuous configuration space for optimal model architectures and training setups. It measures convergence speed and final predictive performance on a binary collision prediction task using pre-extracted dashcam video features.

Datasets

• Nexar dashcam collision prediction dataset — total 1500; splits: train (1200), val (150), test (150)

Metrics

• AP (primary) — range: [0, 1]
  • Average Precision for binary classification, computed as the area under the precision-recall curve. Standard convention for threshold-agnostic evaluation.
• convergence_exponent_c — range: other
  • Exponent c in the power-law model AP*(N) = a - b * N^(-c), fitted to the cumulative best AP over N experiments. Larger c indicates faster convergence.

Input / output format

Input: Natural language text encoding of the experiment history H_{t-1}, including past configurations, validation AP scores, failure diagnostics, and idea genealogy, plus the task description.

Output: Structured YAML configuration specifying discrete choices (backbone, encoder, pooling, loss type, scheduler) and continuous/hyperparameter values (learning rate, weight decay, batch size, sequence length, epochs, oversampling ratio, mixup alpha, feature noise sigma).

Scoring recipe

def compute_ap(y_true, y_prob):
    precisions, recalls, _ = precision_recall_curve(y_true, y_prob)
    return np.trapz(precisions, recalls)
# Applied to validation set for search guidance; final AP reported on held-out test set when available.

Common pitfalls

• Validation AP is used for search guidance, but direct leaderboard comparison requires the held-out competition test set which may not be fully evaluated.
• The search space is mixed categorical-continuous; comparing LLM search directly to purely numeric Bayesian methods (TPE) without accounting for architectural reasoning advantages is misleading.
• Performance variance is dominated by architectural choices (94%) rather than hyperparameter tuning (6%), so focusing solely on LR/batch size optimization misses the primary driver of success.

Evidence (verbatim from paper)

The Nexar collision prediction challenge is an active Kaggle competition; our 0.9245 validation AP provides a strong baseline, though direct leaderboard comparison requires evaluation on the held-out competition test set (which we report when available). We use focal loss to down-weight easy examples and AP as the primary metric following the original challenge evaluation.

Citation

@misc{li2026autoresearching,
  title={Auto Researching, not hyperparameter tuning: Convergence Analysis of 10,000 Experiments},
  author={Li (2026)},
  year={2026},
  note={arXiv:2603.15916}
}

• arXiv: 2603.15916