Swapnet System Eval

swapnet-system-eval

SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget — Wang et al. (2024) (arXiv:2401.16757, 2024)

What this evaluates

Evaluates a block-swapping middleware for DNN inference on memory-constrained edge AI devices. It probes the system's ability to run large models beyond hardware memory limits while measuring peak memory consumption, inference latency, and classification accuracy compared to direct execution, channel division, and model compression baselines across three real-world application scenarios.

Datasets

• GTSRB — total ?; splits: train (-1), test (-1)
• CIFAR100 — total ?; splits: train (-1), test (-1)
• COCO — total ?; splits: train (-1), val (-1)

Metrics

• memory consumption (primary) — range: MB
  • Peak memory usage (MB) required to execute a DNN model or multi-model scenario, measured during inference including OS and non-DNN task overhead.
• latency — range: ms
  • Average inference time (ms) per task or model execution, measured end-to-end from input processing to output generation.
• accuracy — range: percent
  • Standard classification accuracy (e.g., top-1) of the DNN model on its respective test dataset.

Input / output format

Input: Pre-trained DNN model parameters partitioned into blocks (stored on NVMe SSD), real-time sensor data (images/video streams from cameras/LiDAR), and concurrent OS/non-DNN task workloads.

Output: Inference predictions (object detection, segmentation, classification), peak memory consumption (MB), average inference latency (ms), and classification accuracy (%).

Scoring recipe

def evaluate_scenario(scenario, models, baselines):
    metrics = {}
    for model in models:
        peak_mem = measure_peak_memory(model, scenario.budget)
        avg_lat = measure_avg_latency(model, scenario.budget)
        acc = compute_accuracy(model, scenario.test_set)
        metrics[model] = {'memory consumption': peak_mem, 'latency': avg_lat, 'accuracy': acc}
    for baseline in baselines:
        metrics[baseline] = run_baseline(baseline, scenario, models)
    return metrics

Common pitfalls

• Assuming model compression (TPrg) preserves accuracy; the paper explicitly notes it causes 5.0–6.7% accuracy drops, whereas SwapNet maintains original accuracy.
• Ignoring non-DNN task memory overhead when calculating DNN memory budgets; the protocol requires measuring OS/CUDA/SLAM overhead first to determine the true available DNN memory.
• Treating latency as purely a function of block count; partition position and device I/O characteristics significantly impact swapping overhead.

Evidence (verbatim from paper)

Fig.11 compares the memory consumption of each model using the three methods... Fig.11 compares latency... Fig.11 shows that each model with SwapNet can maintain the same high accuracy as in DInf.

Citation

@misc{wang2024swapnet,
  title={SwapNet: Efficient Swapping for DNN Inference on Edge AI Devices Beyond the Memory Budget},
  author={Wang et al. (2024)},
  year={2024},
  note={arXiv:2401.16757}
}

• arXiv: 2401.16757