Dedelayed Eval

dedelayed-eval

Dedelayed: Deleting remote inference delay via on-device correction — Jacobellis et al. (2025) (arXiv:2510.13714, 2025)

What this evaluates

Evaluates a split-inference system's ability to perform real-time semantic segmentation on driving video streams while compensating for simulated network communication delays. It probes temporal prediction capabilities and feature fusion under latency constraints.

Datasets

• BDD100K — total ?; splits: test (-1)

Metrics

• mIoU (primary) — range: [0, 1]
  • Mean Intersection over Union: the average IoU computed per class across all evaluation frames. IoU for a class is the number of correctly predicted pixels divided by the union of ground truth and predicted pixels for that class.

Input / output format

Input: Sequential video frames (compressed via WebP at quality 85) and a simulated network delay parameter τ (0–5 frames / 0–165 ms) fed into the remote model.

Output: Per-frame semantic segmentation masks over 19 Cityscapes classes.

Scoring recipe

def compute_miou(pred_masks, gt_masks, num_classes=19):
    ious = []
    for c in range(num_classes):
        pred_c = (pred_masks == c)
        gt_c = (gt_masks == c)
        intersection = np.logical_and(pred_c, gt_c).sum()
        union = np.logical_or(pred_c, gt_c).sum()
        ious.append(intersection / union if union > 0 else 1.0)
    return np.mean(ious)

Common pitfalls

• Ground truth labels are not provided for all frames; pseudo-labels generated by a pretrained EoMT model are used instead, which may introduce label noise.
• Evaluation simulates delay by feeding future frames with a delay parameter τ sampled uniformly from 0–5 frames, which differs from real-world variable network jitter.
• Results are reported on a subset of 19 Cityscapes classes rather than the full BDD100K label set.

Evidence (verbatim from paper)

On BDD100K, it achieves 6.4 mIoU and 9.8 mIoU gains over local-only and remote-only baselines at 100ms delay, with no added latency, and outperforms both in high-motion scenarios due to effective temporal modeling and mixed-resolution processing. We demonstrate that delay-aware feature fusion can mitigate remote inference latency, sustaining accuracy even when remote predictions are delayed by long communication network latencies.

Citation

@misc{jacobellis2025dedelayed,
  title={Dedelayed: Deleting remote inference delay via on-device correction},
  author={Jacobellis et al. (2025)},
  year={2025},
  note={arXiv:2510.13714}
}

• arXiv: 2510.13714