Openml Cc18 Eval

openml-cc18-eval

OpenML Benchmarking Suites — Bischl et al. (2017) (arXiv:1708.03731, 2017)

What this evaluates

Evaluates machine learning classifiers on a curated collection of standardized classification tasks. It probes the reproducibility and comparability of algorithm performance across diverse datasets under consistent, machine-readable evaluation protocols.

Datasets

• OpenML-CC18 — total ?; splits: train (-1), test (-1); repo https://github.com/openml/benchmark-suites

Metrics

• accuracy_score (primary) — range: [0, 1]
  • Standard classification accuracy: the proportion of correctly predicted instances out of the total number of instances in the evaluation set.

Input / output format

Input: An OpenML task object containing a dataset, predefined train/test splits, and evaluation instructions, along with a machine learning learner or pipeline configuration.

Output: A run object containing the computed metric score (accuracy) for the task, which can be published to the OpenML platform for sharing and comparison.

Scoring recipe

benchmark_suite = study.get_suite('OpenML-CC18')
for task_id in benchmark_suite.tasks:
    task = tasks.get_task(task_id)
    run = runs.run_model_on_task(clf, task)
    score = run.get_metric_fn(metrics.accuracy_score)
    print(f'Accuracy: {score.mean():.2f}')

Common pitfalls

• Confusing the general OpenML dataset repository with the curated OpenML-CC18 benchmarking suite.
• Assuming static train/test splits; OpenML tasks define splits programmatically, so results depend on the exact task configuration.
• Overlooking that the suite is dynamic and can be extended or corrected over time, meaning benchmarks are not permanent snapshots.

Evidence (verbatim from paper)

An OpenML benchmarking suite is a set of OpenML tasks carefully selected to evaluate algorithms under a precise set of conditions. score = run.get_metric_fn(metrics.accuracy_score) # compute and print the accuracy score

Citation

@misc{bischl2017openml,
  title={OpenML Benchmarking Suites},
  author={Bischl et al. (2017)},
  year={2017},
  note={arXiv:1708.03731}
}

• arXiv: 1708.03731