Task Bench Eval

task-bench-eval

Task Bench: A Parameterized Benchmark for Evaluating Parallel Runtime Performance — Slaughter et al. (2019) (arXiv:1908.05790, 2019)

What this evaluates

This benchmark evaluates the parallel runtime performance and scalability of different programming systems by executing parameterized task graphs. It probes how efficiently systems handle varying degrees of parallelism, communication patterns, and computational versus memory-bound workloads.

Datasets

• Task Bench — total ?; splits: default (-1)

Metrics

• minimum effective task granularity (METG) (primary) — range: microseconds
  • The smallest task size (in microseconds) at which a parallel runtime system maintains 50% efficiency compared to its peak performance. Calculated by sweeping task duration/iterations and measuring execution time to find the efficiency breakpoint.

Input / output format

Input: A parameterized task graph configuration specifying height (timesteps), width (parallelism), dependence pattern (e.g., stencil, FFT, tree), kernel type (compute or memory-bound), iterations (task duration), span/scratch (memory size), and load imbalance degree.

Output: Execution time (runtime) for the task graph, plus a validation tuple ⟨row, col⟩ per task to verify correctness against expected dependencies.

Scoring recipe

def score_task_bench(config, runtime_impl):
    graph = TaskBench.generate(config)
    start_time = measure_time()
    runtime_impl.execute(graph)
    end_time = measure_time()
    runtime = end_time - start_time
    for task in graph.tasks:
        assert task.get_output() == compute_expected_output(config, task.id)
    peak_runtime = get_peak_runtime(config)
    efficiency = peak_runtime / runtime
    metg = find_breakpoint(config.iterations, efficiency)
    return {'runtime': runtime, 'efficiency': efficiency, 'METG': metg}

Common pitfalls

• Cache effects can artificially speed up smaller tasks if the working set size isn't kept constant across iterations.
• Not all runtime systems automatically exploit task parallelism; some (like basic MPI CSP patterns) may leave performance on the table under load imbalance.
• Validation overhead (checking dependencies) can skew results at the smallest task granularities, though it is kept under 3%.

Evidence (verbatim from paper)

The framework enables a novel performance metric—minimum effective task granularity (METG)—which quantifies the smallest task size at which a system maintains 50% efficiency, revealing that current technologies have a practical lower bound of 100 µs for scalable performance, despite variations in overhead due to node count, accelerators, and dependency complexity.

Citation

@misc{slaughter2019taskbench,
  title={Task Bench: A Parameterized Benchmark for Evaluating Parallel Runtime Performance},
  author={Slaughter et al. (2019)},
  year={2019},
  note={arXiv:1908.05790}
}

• arXiv: 1908.05790