SQL Hadoop Comparison Eval

sql-hadoop-comparison-eval

A comparative analysis of state-of-the-art SQL-on-Hadoop systems for interactive analytics — Tapdiya et al. (2018) (arXiv:1804.00224, 2018)

What this evaluates

This evaluation compares the interactive analytics performance of four SQL-on-Hadoop systems (Impala, Drill, Spark SQL, Phoenix) by measuring query response times and resource utilization. It characterizes how each system's optimizer and execution engine handle join orders, operator selection, and data scanning across different storage formats and scaling configurations.

Datasets

• Unspecified SQL workloads (text/parquet) — total ?; splits: (unstated)

Metrics

• query_rt (primary) — range: other
  • Query Response Time (RT), measured as the total wall-clock time to execute a SQL query from submission to result retrieval.

Input / output format

Input: SQL queries executed against four SQL-on-Hadoop systems (Impala, Drill, Spark SQL, Phoenix) on datasets stored in either text (row-wise) or Parquet (columnar) formats.

Output: Query response time (RT), resource utilization metrics, and query execution profiles containing per-operator processing times.

Scoring recipe

def compute_metrics(predictions, gold):
    system_rts = {}
    for sys, runs in predictions.items():
        rts = [run['rt'] for run in runs]
        system_rts[sys] = sum(rts) / len(rts)
    
    operator_contributions = {}
    for sys, runs in predictions.items():
        for run in runs:
            for op, time in run['operator_times'].items():
                operator_contributions.setdefault(sys, {}).setdefault(op, 0)
                operator_contributions[sys][op] += time
    return system_rts, operator_contributions

Common pitfalls

• The evaluation focuses on system-level bottlenecks (CPU, Disk, Network, Memory) rather than query correctness, so exact-match validation is secondary to RT profiling.
• Scale-up and size-up experiments require strict proportional scaling of cluster nodes and data size; deviating from this breaks the comparability of linear vs. sub-linear scaling claims.

Evidence (verbatim from paper)

The query RT represents our performance metric. We collect resource utilization metrics and query execution profiles in each system. In each system, we aggregate the processing times for each operator type to understand the contribution of each operator type to the query RT.

Citation

@misc{tapdiya2018comparative,
  title={A comparative analysis of state-of-the-art SQL-on-Hadoop systems for interactive analytics},
  author={Tapdiya et al. (2018)},
  year={2018},
  note={arXiv:1804.00224}
}

• arXiv: 1804.00224