M Design Model Selection Eval

m-design-model-selection-eval

Beyond Model Base Selection: Weaving Knowledge to Master Fine-grained Neural Network Design — Wang et al. (2025) (arXiv:2507.15336, 2025)

What this evaluates

Evaluates the effectiveness and refinement efficiency of neural network architecture search and selection methods on graph datasets. It measures how well a method can find near-optimal models within a limited search budget and how quickly it reaches a target performance level across diverse graph topologies and tasks.

Datasets

• Graph Architecture Search Benchmark (22 datasets) — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• classification accuracy / AUC-ROC (primary) — range: [0, 1]
  • Proportion of correctly classified instances (accuracy) or area under the receiver operating characteristic curve (AUC-ROC), depending on the task type. Evaluated under a maximum search budget of 100 model-testings.
• number of model-testings — range: integer
  • The count of model evaluations required to reach a predefined target performance level. Capped at 100 if the target is not met within the search budget.
• Kendall rank correlation — range: [-1, 1]
  • Measures the ordinal association between predicted and actual local modification consistency rankings, used in ablation studies.

Input / output format

Input: Graph datasets with node features, edges, and labels, along with a predefined model design space and task metadata.

Output: Selected or refined neural network architecture, and its performance metrics (accuracy/AUC-ROC) after up to 100 model-testings.

Scoring recipe

def compute_metrics(predictions, gold_labels, target_perf, max_budget=100):
    accuracy = sum(1 for p, g in zip(predictions, gold_labels) if p == g) / len(gold_labels)
    iterations = max_budget
    for i, perf in enumerate(predictions):
        if perf >= target_perf:
            iterations = i + 1
            break
    return {'accuracy': accuracy, 'model_testings': iterations}

Common pitfalls

• Search budget is strictly capped at 100 model-testings; methods failing to reach the target performance within this limit are recorded as 100.
• Target performance is task-specific, defined as the halfway point of M-DESIGN's 50-iteration run, not a fixed global threshold.
• All results are averaged over 10 repeated trials; single-run evaluations will deviate significantly from reported numbers.

Evidence (verbatim from paper)

We evaluate the model selection (1) effectiveness with the classification accuracy or AUC-ROC on all data-task pairs under a maximum search budget of 100 model-testings and (2) refinement efficiency with the number of model-testings (i.e., the basic time unit in the benchmarking setting) required for methods to achieve the target performance level, i.e., the halfway performance marked by running 50 iterations of M-DESIGN on the node classification task.

Citation

@misc{wang2025mdesign,
  title={Beyond Model Base Selection: Weaving Knowledge to Master Fine-grained Neural Network Design},
  author={Wang et al. (2025)},
  year={2025},
  note={arXiv:2507.15336}
}

• arXiv: 2507.15336