Microisp Eval

microisp-eval

MicroISP: Processing 32MP Photos on Mobile Devices with Deep Learning — Ignatov et al. (2022) (arXiv:2211.06770, 2022)

What this evaluates

Evaluates a deep learning-based image signal processing (ISP) model's ability to reconstruct high-resolution RGB images from RAW mobile sensor data. It measures reconstruction fidelity, visual quality, and inference efficiency across various mobile hardware platforms and resolutions.

Datasets

• Fujifilm UltraISP — total ?; splits: test (-1)

Metrics

• PSNR (primary) — range: dB
  • Peak Signal-to-Noise Ratio, calculated as 10 * log10(MAX_I^2 / MSE), where MAX_I is the maximum possible pixel value and MSE is the mean squared error between reconstructed and target RGB images.
• SSIM — range: [0, 1]
  • Structural Similarity Index Measure, evaluating luminance, contrast, and structure similarity between reconstructed and target images.
• Runtime — range: ms
  • Inference latency measured in milliseconds per image on specific mobile hardware using the AI Benchmark application.

Input / output format

Input: RAW sensor images (e.g., 32MP Sony sensor RAW data)

Output: Reconstructed RGB images

Scoring recipe

def compute_metrics(preds, gold):
    mse = np.mean((preds - gold) ** 2)
    psnr = 10 * np.log10((255.0**2) / mse)
    ssim = calculate_ssim(preds, gold)
    return {'PSNR': psnr, 'SSIM': ssim}

def measure_runtime(model, input_tensor, device):
    for _ in range(10): model(input_tensor) # warmup
    start = time.perf_counter()
    model(input_tensor)
    end = time.perf_counter()
    return (end - start) * 1000 # ms

Common pitfalls

• Runtime results are highly hardware-dependent and reported only for specific mobile SoCs (e.g., MediaTek Dimensity 1000+ GPU); results will not generalize to other devices without re-evaluation.
• PSNR and SSIM scores do not always correlate with perceptual quality; hand-crafted ISP pipelines often produce visually 'better' images due to sharpening and color boosting despite lower numerical scores.
• Memory constraints (OOM) heavily limit the maximum processable resolution for baseline models, making runtime comparisons at high resolutions (26MP/32MP) impossible for many baselines.

Evidence (verbatim from paper)

All models were trained on the Fujifilm UltraISP dataset, their PSNR and SSIM scores on the test image subset are reported in Table[1], sample visual results for all methods are demonstrated in Fig.[5]. The proposed MicroISP network was able to substantially outperform the other solutions in almost all aspects. In particular, it offers a 0.5dB PSNR improvement compared to the baseline FSRCNN model, and outperforms by 0.14 dB the CSANet model demonstrating the second best PSNR results on the considered dataset.

Citation

@misc{ignatov2022microisp,
  title={MicroISP: Processing 32MP Photos on Mobile Devices with Deep Learning},
  author={Ignatov et al. (2022)},
  year={2022},
  note={arXiv:2211.06770}
}

• arXiv: 2211.06770