qhjqhj00 peernet-profiling-eval
PEERNet: An End-to-End Profiling Tool for Real-Time Networked Robotic Systems — Narayanan et al. (2024) (arXiv:2409.06078, 2024)
What this evaluates
Evaluates a profiling framework's ability to measure granular, end-to-end latency and network asymmetry across heterogeneous hardware and live wireless networks in robotic systems. It probes how well the tool captures component-level timing, inference variance, and transmission delays in real-world deployments.
Datasets
- ImageNet — total ?; splits: test (-1)
- Waymo Open Dataset — total ?; splits: test (-1)
- Franka Emika Panda Teleoperation Setup — total ?; splits: test (-1)
Metrics
end-to-end latency(primary) — range: ms- Sum of sensing, upload, inference, and download latencies per step. Reported as mean ± standard deviation over valid steps.
upload latency— range: ms- Time taken to transmit images from edge to cloud device over the network.
download latency— range: ms- Time taken to transmit control actions from cloud back to edge device.
inference latency— range: ms- Time taken by the GPU to process the input and generate output.
Input / output format
Input: Image/video frames, text prompts, and robot state observations.
Output: Latency breakdown per pipeline stage (sensing, upload, inference, download, total) with mean ± std over episodes.
Scoring recipe
def compute_latency_metrics(steps_data):
# steps_data: list of lists of dicts with keys 'sensing', 'upload', 'inference', 'download', 'total'
# Discard first episode and first step per episode for warm-up
valid_steps = [s for ep_idx, ep in enumerate(steps_data)
for s_idx, s in enumerate(ep) if ep_idx > 0 or s_idx > 0]
metrics = {}
for key in valid_steps[0].keys():
vals = [s[key] for s in valid_steps]
mean = sum(vals) / len(vals)
std = (sum((x - mean)**2 for x in vals) / len(vals))**0.5
metrics[key] = f"{mean:.2f} ± {std:.2f}"
return metrics
Common pitfalls
- Failing to discard the first episode and first step, which skews measurements due to GPU warm-up.
- Assuming symmetric network delays, whereas upload and download latencies can differ significantly.
- Ignoring inference latency variance, which is often higher on cloud servers due to multi-tenancy.
Evidence (verbatim from paper)
PEERNet quantifies the tradeoffs between network latency and inference cost, disambiguates upload and download latency, and identifies the most performative setups.
Citation
@misc{narayanan2024peernet,
title={PEERNet: An End-to-End Profiling Tool for Real-Time Networked Robotic Systems},
author={Narayanan et al. (2024)},
year={2024},
note={arXiv:2409.06078}
}
- arXiv: 2409.06078