Bigop Eval

bigop-eval

BigOP: Generating Comprehensive Big Data Workloads as a Benchmarking Framework — Zhu et al. (2014) (arXiv:1401.6628, 2014)

What this evaluates

Evaluates big data processing systems under diverse workload patterns (record insertion, statistics computation, iterative graph computation) to measure throughput, latency, and execution time across relational, text, and graph data types.

Datasets

• BigOP Log Monitoring & PageRank Workloads — total ?; splits: test (-1)

Metrics

• throughput (ops/sec) (primary) — range: other
  • Total number of operations completed divided by the total execution time.
• latency (us) — range: other
  • Time elapsed per operation, reported as minimum, maximum, and average values in microseconds.
• run_time (sec) — range: other
  • Total wall-clock time required to complete a task or query, reported as minimum and maximum across runs.

Input / output format

Input: Prescribed workload specifications defining operations, patterns, data size, and velocity, applied to target big data systems (MySQL Cluster, Hadoop+HBase, HDFS+Spark).

Output: Performance measurements including throughput (ops/sec), latency statistics (min, max, average in microseconds), and execution time (seconds).

Scoring recipe

# Calculate throughput
throughput = total_operations / total_execution_time
# Calculate latency per operation
latency_us = operation_timestamps_diff * 1_000_000
# Aggregate latency
min_latency = min(latency_us)
max_latency = max(latency_us)
avg_latency = sum(latency_us) / len(latency_us)
# Report run time
run_time_sec = total_execution_time

Common pitfalls

• Hardware setup heavily influences results (5 nodes, 1Gbps network, 32GB RAM, 6-core 2.4GHz CPUs).
• MySQL Cluster's batch commit mode can report 0 minimum latency, which is an artifact of the commit strategy rather than true operation speed.
• System performance is highly workload-dependent; e.g., Hadoop+HBase excels at record inserts while MySQL Cluster dominates statistics computation.

Evidence (verbatim from paper)

Log Monitoring. Figure 3 demonstrates the resulting performances of SUT1 and SUT2. SUT2 excels under the frequent record insert task as expected. SUT1 performs much better in statistics computation tasks because of the long starting time of jobs in SUT2. ... Throughput (ops/sec) | Min Latency (us) | Max Latency (us) | Average Latency (us) ... The running time of the task over SUT2 is 363 seconds, while that over SUT3 is 96 seconds.

Citation

@misc{zhu2014bigop,
  title={BigOP: Generating Comprehensive Big Data Workloads as a Benchmarking Framework},
  author={Zhu et al. (2014)},
  year={2014},
  note={arXiv:1401.6628}
}

• arXiv: 1401.6628