Split Computing Eval

split-computing-eval

A Multi-task Supervised Compression Model for Split Computing — Matsubara et al. (2025) (arXiv:2501.01420, 2025)

What this evaluates

Evaluates a multi-task supervised compression model for split computing across image classification, object detection, and semantic segmentation. It measures predictive accuracy alongside system-level metrics like end-to-end latency and energy consumption on resource-constrained edge devices with simulated wireless links.

Datasets

• ILSVRC 2012 — total ?; splits: test (-1)
• COCO 2017 — total ?; splits: test (-1)
• PASCAL VOC 2012 — total ?; splits: test (-1)

Metrics

• encoder size — range: other
  • Total size of the shared lightweight encoder parameters, typically calculated as the number of parameters multiplied by 4 bytes (assuming FP32 precision).
• compressed data size — range: other
  • Total number of bytes transmitted over the wireless link for the encoded representation, measured in megabytes (MB).
• model accuracy (primary) — range: [0, 1]
  • Standard predictive accuracy for each task: Top-1 accuracy for image classification, mean Average Precision (mAP) for object detection, and mean Intersection over Union (mIoU) for semantic segmentation.
• end-to-end latency — range: other
  • Total time from input image ingestion to final task prediction output, including encoding, transmission over the simulated wireless link, and server-side decoding/inference.
• energy consumption — range: other
  • Total energy in Joules consumed by the mobile device and edge server during inference, calculated by integrating instantaneous power readings using Simpson’s rule.

Input / output format

Input: Raw images from ILSVRC 2012, COCO 2017, or PASCAL VOC 2012, passed through a shared lightweight encoder for split computing and transmitted over a simulated wireless link (100 Kbps or LoRa at 37.5 Kbps) to an edge server.

Output: Task-specific predictions (class labels, bounding boxes, or segmentation masks) and system-level measurements (end-to-end latency in seconds, energy consumption in Joules).

Scoring recipe

def compute_metrics(predictions, gold, power_trace, start_time, end_time, encoder, transmissions):
    acc = compute_task_accuracy(predictions, gold)  # Top-1/mAP/mIoU
    latency = end_time - start_time
    energy = simpson_integrate(power_trace)
    encoder_size = encoder.num_parameters() * 4
    compressed_size = sum(len(p) for p in transmissions)
    return {'accuracy': acc, 'latency': latency, 'energy': energy, 'encoder_size': encoder_size, 'compressed_size': compressed_size}

Common pitfalls

• Failing to warm up devices before measuring latency and power consumption, leading to unstable GPU behavior and inaccurate readings.
• Confusing encoder size (model parameters) with compressed data size (transmitted payload), which are distinct metrics in split computing.
• Ignoring the specific wireless data rate (100 Kbps vs 37.5 Kbps) when calculating transmission latency, which significantly impacts end-to-end timing.

Evidence (verbatim from paper)

Our evaluations are based on the benchmark framework for split computing in[[24]], which adopts three key metrics: 1) encoder size, 2) compressed data size, and 3) model accuracy. Additionally, we consider end-to-end latency and energy consumption of mobile devices in multi-task scenarios.

Citation

@misc{matsubara2025multitask,
  title={A Multi-task Supervised Compression Model for Split Computing},
  author={Matsubara et al. (2025)},
  year={2025},
  note={arXiv:2501.01420}
}

• arXiv: 2501.01420