Mai Aim2022 Depth Eval

mai-aim2022-depth-eval

LiteDepth: Digging into Fast and Accurate Depth Estimation on Mobile Devices — Li et al. (2022) (arXiv:2209.00961, 2022)

What this evaluates

Evaluates the accuracy and inference speed of monocular depth estimation models on resource-constrained mobile devices. It measures depth prediction quality using invariant standard root mean squared error and records inference time on a Raspberry Pi 4 to assess real-time capability.

Datasets

• MAI&AIM2022 challenge dataset — total 7385; splits: train (6869), val (516)

Metrics

• si-RMSE (primary) — range: other
  • Invariant standard root mean squared error. Computes the RMSE of the log-depth difference after removing the mean shift between prediction and ground truth, making it scale- and shift-invariant.
• runtime — range: other
  • Inference time measured in milliseconds on a Raspberry Pi 4 using TFLite.
• Score — range: other
  • Score(si-RMSE, runtime) = (2^{-20} * si-RMSE) / (C * runtime), where C=0.01 on the online validation benchmark. Higher values indicate better combined accuracy and speed.

Input / output format

Input: RGB image (640x480 resolution, float32)

Output: Predicted depth map (640x480 resolution, uint16 format representing 0-40 meters)

Scoring recipe

def compute_si_rmse(pred, gt):
    log_pred = np.log(pred)
    log_gt = np.log(gt)
    diff = log_pred - log_gt - np.mean(log_pred - log_gt)
    return np.sqrt(np.mean(diff**2))

def compute_score(si_rmse, runtime_ms):
    C = 0.01
    return (2**-20 * si_rmse) / (C * runtime_ms)

Common pitfalls

• Runtime must be measured on a Raspberry Pi 4 using TFLite, not on desktop GPUs or other hardware.
• Depth maps are stored as uint16 values representing 0-40 meters; incorrect scaling or type conversion will break the metric.
• The scoring formula uses a constant C=0.01 specifically for the online validation benchmark, which directly impacts the final ranking.

Evidence (verbatim from paper)

In MAI&AIM2022 challenge, two metrics are considered for each submission solution: 1) The quality of the depth estimation. It is measured by the invariant standard root mean squared error (si-RMSE). 2) The runtime of the model on the target platform (i.e., Raspberry Pi 4). The scoring formulation is provided below: Score(si-RMSE, runtime) = (2^{-20} * si-RMSE) / (C * runtime), where C=0.01 on the online validation benchmark.

Citation

@misc{li2022litedepth,
  title={LiteDepth: Digging into Fast and Accurate Depth Estimation on Mobile Devices},
  author={Li et al. (2022)},
  year={2022},
  note={arXiv:2209.00961}
}

• arXiv: 2209.00961