Workflow Benchmark Accuracy Eval

workflow-benchmark-accuracy-eval

WfBench: Automated Generation of Scientific Workflow Benchmarks — Coleman et al. (2022) (arXiv:2210.03170, 2022)

What this evaluates

Evaluates whether automatically generated scientific workflow benchmarks accurately replicate the execution time and performance characteristics of real scientific workflows under varying hardware architectures and external memory loads.

Datasets

• Montage — total ?; splits: (unstated)
• 1000Genome — total ?; splits: (unstated)

Metrics

• execution_time_ratio (primary) — range: other
  • Ratio of benchmark execution time to real workflow execution time ($T_{bench} / T_{real}$). Values close to 1.0 indicate high accuracy.
• makespan_percentage_difference — range: percent
  • Percentage difference in total workflow makespan between the generated benchmark and the real workflow: $((T_{bench} - T_{real}) / T_{real}) \times 100$.

Input / output format

Input: Real workflow task executables (e.g., Montage, 1000Genome), compute node hardware specifications (CPU, cores, RAM), and external load conditions (e.g., stress-ng instances).

Output: Generated benchmark configuration (JSON), execution times for both the real workflow task and the instantiated benchmark, and the computed ratio/difference.

Scoring recipe

def compute_accuracy(real_time, bench_time):
    ratio = bench_time / real_time
    pct_diff = ((bench_time - real_time) / real_time) * 100
    return ratio, pct_diff

Common pitfalls

• Confusing the configuration parameter $f$ (fraction of CPU instructions) with the evaluation metric itself.
• Assuming benchmark accuracy generalizes across hardware architectures without empirical validation under varying memory loads.
• Ignoring the impact of rounding $f$ to multiples of 0.1, which can cause significant deviations in concurrent execution scenarios.

Evidence (verbatim from paper)

Each plot is for different Montage and 1000Genome tasks and shows the ratio between the benchmark’s execution time and that of the workflow task (vertical axis) vs. $f$ (horizontal axis). Values above (resp. below) 1.0 correspond to cases in which the benchmark execution is longer (resp. shorter) than that of the workflow task.

Citation

@misc{coleman2022wfbench,
  title={WfBench: Automated Generation of Scientific Workflow Benchmarks},
  author={Coleman et al. (2022)},
  year={2022},
  note={arXiv:2210.03170}
}

• arXiv: 2210.03170