Superbench Eval

superbench-eval

SuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validation — Xiong et al. (2024) (arXiv:2402.06194, 2024)

What this evaluates

Evaluates the effectiveness of a proactive validation system for cloud AI infrastructure in detecting hardware defects, selecting optimal benchmark subsets, and balancing validation cost against system reliability. It probes the ability of automated criteria and selection algorithms to distinguish healthy nodes from degraded ones while minimizing downtime and maximizing GPU utilization.

Datasets

• Cluster Benchmark Dataset — total 3000; splits: test (3000); repo https://github.com/microsoft/superbenchmark

Metrics

• Margin Ratio (primary) — range: [0, 1]
  • Calculated as the minimum distance between any defective node's score and the criteria vector, divided by the maximum distance between any healthy node's score and the criteria vector. Higher values indicate a clearer separation between healthy and defective nodes.
• Repeatability — range: [0, 1]
  • Measured as the arithmetic mean of similarity scores between each benchmark sample and the derived criteria vector. Values closer to 1 indicate higher consistency across runs.
• MTBI — range: other
  • Mean Time Between Incidents, calculated by dividing a node's total up time by the number of incidents that occurred within that period.

Input / output format

Input: Benchmark step throughput metrics for each VM across 24 benchmarks; node status features (uptime, historical incident counts, category-specific MTBI) for probability modeling.

Output: Binary defect classification (healthy/defective) per node based on criteria thresholds; predicted Time Before Next Incident (TBNI) for probability models; selected subset of benchmarks for validation.

Scoring recipe

def compute_margin_ratio(healthy_scores, defective_scores, criteria_vector):
    dist_healthy = [distance(s, criteria_vector) for s in healthy_scores]
    dist_defective = [distance(s, criteria_vector) for s in defective_scores]
    min_def = min(dist_defective)
    max_healthy = max(dist_healthy)
    return min_def / max_healthy if max_healthy > 0 else 0

def compute_repeatability(samples, criteria_vector):
    similarities = [similarity(s, criteria_vector) for s in samples]
    return sum(similarities) / len(similarities)

Common pitfalls

• No ground-truth labels exist for defective nodes; defects are operationally defined by observed failures or performance regressions, which may miss subtle gray failures.
• The criteria vector S_C is derived from the same dataset used for evaluation, potentially inflating margin ratios if the clustering method overfits to the training distribution.
• MTBI and utilization metrics are highly sensitive to simulation assumptions, such as fixed repair times (1.5 days vs 1 hour) and incident category distributions.

Evidence (verbatim from paper)

To compare whether the criteria can maximize the margin between healthy and defective nodes, we define the Margin Ratio metric as $\frac{\min_{i\in\text{defective}}d(S_i,S_C)}{\max_{j\in\text{healthy}}d(S_j,S_C)}$ Apart from the proposed method, we establish two baselines with typical outlier detection methods, including the interquartile range (IQR) [20] and $k$ -means [26].

Citation

@misc{xiong2024superbench,
  title={SuperBench: Improving Cloud AI Infrastructure Reliability with Proactive Validation},
  author={Xiong et al. (2024)},
  year={2024},
  note={arXiv:2402.06194}
}

• arXiv: 2402.06194