Scientific Image Analytics Eval

scientific-image-analytics-eval

Comparative Evaluation of Big-Data Systems on Scientific Image Analytics Workloads — Mehta et al. (2016) (arXiv:1612.02485, 2016)

What this evaluates

Evaluates the ease of implementation and qualitative complexity of five big-data systems when executing real-world scientific image analytics workloads in astronomy and neuroscience. It probes how well each system handles multidimensional array operations, Python integration, and native support for scientific image formats.

Datasets

• Neuroscience Use Case — total ?; splits: (unstated)
• Astronomy Use Case — total ?; splits: (unstated)

Metrics

• Lines of Code (LoC) (primary) — range: other
  • Count of source code lines required to implement each pipeline step (e.g., data ingest, segmentation, denoising) in the target system's native API or query language. Excludes boilerplate and reference code reuse.
• Qualitative Complexity Assessment — range: other
  • Author-provided subjective evaluation of implementation difficulty, debuggability, and required physical tuning for each system.

Input / output format

Input: Scientific image data files (NIfTI for neuroscience, FITS for astronomy) and reference Python implementations defining the pipeline steps.

Output: Implementation code in the target system's native language/API (e.g., AQL/AFL, Spark RDDs, MyriaL, Dask delayed graphs, TensorFlow graphs) and the total line count per pipeline step.

Scoring recipe

def compute_loc(system_code, steps):
    loc_per_step = {}
    for step in steps:
        lines = [l for l in system_code[step].split('\n') if l.strip() and not l.strip().startswith('#')]
        loc_per_step[step] = len(lines)
    return loc_per_step

Common pitfalls

• LoC counts heavily favor systems that allow reusing reference Python code (e.g., Spark, Myria) over those requiring full rewrites in native APIs (e.g., TensorFlow, SciDB).
• Qualitative assessments of debuggability and tuning effort are subjective and not quantified, making cross-system comparisons partially dependent on author interpretation.
• Performance and runtime metrics are explicitly deferred to a separate section, so this evaluation only measures implementation complexity, not execution efficiency.

Evidence (verbatim from paper)

The first dimension, which we present in this section, is the system’s ease of use, which we measure using lines of code (LoC) needed to implement the use cases and a qualitative assessment of overall implementation complexity.

Citation

@misc{mehta2016comparative,
  title={Comparative Evaluation of Big-Data Systems on Scientific Image Analytics Workloads},
  author={Mehta et al. (2016)},
  year={2016},
  note={arXiv:1612.02485}
}

• arXiv: 1612.02485