qhjqhj00 cloud-vm-ranking-eval
Cloud Benchmarking for Performance — Varghese et al. (2014) (arXiv:1411.0912, 2014)
What this evaluates
This benchmark evaluates how effectively different Virtual Machines (VMs) can execute specific application workloads by ranking them according to weighted hardware attributes. It probes the capability to map domain-specific application requirements to underlying infrastructure performance characteristics.
Datasets
- Cloud VM Benchmarking Suite — total ?; splits: (unstated)
Metrics
S_i(primary) — range: other- S_i = \bar{G}{i,k} \cdot W_k, where \bar{G}{i,k} is the normalized attribute group for VM i, and W_k is the user-provided weight (0–5) for group k.
Input / output format
Input: Attribute values r_{i,j} for each VM, grouped into categories (memory, processor, computation, storage), and user-provided weights W_k (0–5) for each group.
Output: A ranked list of VMs (Rp_i) ordered by descending performance score S_i.
Scoring recipe
# Normalize attributes across all candidate VMs
r_bar = (r - mean(r, axis=0)) / std(r, axis=0)
# Compute weighted score per VM
S = sum(r_bar[i, k] * W[k] for k in groups)
# Rank VMs by score descending
Rp = argsort(-S)
Common pitfalls
- Normalization is performed relative to the set of candidate VMs, not a fixed baseline, so rankings are pool-dependent.
- Weights must be explicitly provided by domain experts based on application requirements; uniform weights defeat the purpose of application-aware ranking.
- Validation requires empirical execution time comparison against the theoretical ranking, not just score matching.
Evidence (verbatim from paper)
The score of each VM is calculated as $S_{i}=\bar{G}{i,k}.W{k}$. The scores are ordered in a descending order for generating $Rp_{i}$ which is the ranking of the VMs based solely on performance.
Citation
@misc{varghese2014cloudbenchmarking,
title={Cloud Benchmarking for Performance},
author={Varghese et al. (2014)},
year={2014},
note={arXiv:1411.0912}
}
- arXiv: 1411.0912