Fanstore Io Eval

fanstore-io-eval

FanStore: Enabling Efficient and Scalable I/O for Distributed Deep Learning — Zhang et al. (2018) (arXiv:1809.10799, 2018)

What this evaluates

Evaluates the I/O throughput and bandwidth of a distributed runtime file system (FanStore) across varying node counts and file sizes, comparing it against local SSDs, FUSE, and shared file systems like Lustre.

Datasets

• ImageNet-1k — total ?; splits: full (-1)
• SRGAN — total 455000000000; splits: full (-1)
• FRNN — total 54000000000; splits: full (-1)
• Custom Synthetic Benchmark — total ?; splits: full (-1)

Metrics

• bandwidth (MB/s) (primary) — range: other
  • Total bytes read divided by the total time taken to read all files across all nodes.
• throughput (files/s) (primary) — range: other
  • Total number of files read divided by the total time taken to read all files across all nodes.
• scaling efficiency (%) — range: percent
  • Ratio of throughput at N nodes to baseline throughput, normalized by the node count increase.

Input / output format

Input: Distributed directory of files accessed concurrently by multiple compute nodes via POSIX system calls intercepted by FanStore.

Output: Aggregated bandwidth (MB/s) and throughput (files/s) per node scale and file size configuration.

Scoring recipe

def compute_metrics(total_files, total_bytes, elapsed_seconds):
    throughput = total_files / elapsed_seconds
    bandwidth = total_bytes / elapsed_seconds
    return throughput, bandwidth

def compute_scaling_efficiency(throughput_n, throughput_baseline, nodes_n, nodes_baseline):
    return (throughput_n / throughput_baseline) / (nodes_n / nodes_baseline) * 100

Common pitfalls

• Assuming shared file systems (e.g., Lustre) scale linearly; performance actually fluctuates and degrades under high concurrency.
• Ignoring that compression benefits are highly dependent on file size and whether the bottleneck is CPU, network latency, or network bandwidth.
• Treating local SSD performance as the achievable baseline for networked storage, rather than an upper bound.

Evidence (verbatim from paper)

This benchmark has four file sizes: 128 KB, 512 KB, 2 MB, and 8 MB. Each file size has {128K, 32K, 8K, 2K} file count, respectively. At each scale, each node reads all files in the directory, and reports time-to-solution and bandwidth. The benchmark reports the aggregated bandwidth and throughput as results.

Citation

@misc{zhang2018fanstore,
  title={FanStore: Enabling Efficient and Scalable I/O for Distributed Deep Learning},
  author={Zhang et al. (2018)},
  year={2018},
  note={arXiv:1809.10799}
}

• arXiv: 1809.10799