Learned Isp Eval

learned-isp-eval

Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report — Ignatov et al. (2022) (arXiv:2211.03885, 2022)

What this evaluates

Evaluates end-to-end learned image signal processing (ISP) pipelines that map mobile RAW sensor data to high-fidelity RGB images. It probes the trade-off between image reconstruction fidelity, subjective visual quality, and real-time inference efficiency on mobile hardware.

Datasets

• Fujifilm UltraISP dataset — total ?; splits: test (-1)

Metrics

• PSNR (primary) — range: dB
  • Peak Signal-to-Noise Ratio, calculated as 10 * log10(MAX_I^2 / MSE) where MAX_I is the maximum possible pixel value and MSE is the mean squared error between predicted and reference images.
• MOS — range: [1, 5]
  • Mean Opinion Score, a subjective visual quality rating typically on a 1-5 scale, aggregated across human raters.
• Runtime — range: ms
  • Inference latency measured in milliseconds on the target Snapdragon 8 Gen 1 GPU for Full HD resolution.

Input / output format

Input: Mobile RAW sensor data frames

Output: High-fidelity RGB images (Full HD resolution)

Scoring recipe

psnr = calculate_psnr(predictions, ground_truth_rgb)
mos = aggregate_human_ratings(predictions)
runtime_ms = measure_latency_on_snapdragon_8_gen1(predictions, resolution='1920x1080')
# Final ranking combines PSNR, MOS, and runtime; models must pass TFLite compatibility and factsheet checks

Common pitfalls

• Conventional fidelity metrics like PSNR often fail to correlate with perceived real image quality, especially for color rendition and texture rendering.
• Incorrect model conversion (e.g., PyTorch to ONNX to TFLite) can corrupt weights, leading to invalid submissions despite good architecture.
• Optimizing solely for runtime (e.g., using very shallow networks) can significantly degrade visual fidelity and MOS scores.

Evidence (verbatim from paper)

Tables[1] and[2] demonstrate the fidelity, runtime and MOS results of all solutions submitted during the final test phase. Solutions with a MOS score of less than 2.8 were usually having several issues or were exhibiting noticeable image corruptions. These results highlighted again the difficulty of an accurate assessment of the results obtained in learned ISP task as the conventional fidelity metrics are often not indicating the real image quality.

Citation

@misc{ignatov2022learnedsmartphoneisp,
  title={Learned Smartphone ISP on Mobile GPUs with Deep Learning, Mobile AI & AIM 2022 Challenge: Report},
  author={Ignatov et al. (2022)},
  year={2022},
  note={arXiv:2211.03885}
}

• arXiv: 2211.03885