Raw Instinct Eval

raw-instinct-eval

Raw Instinct: Trust Your Classifiers and Skip the Conversion — Kantas et al. (2024) (arXiv:2403.14439, 2024)

What this evaluates

Evaluates whether direct classification of RAW sensor data achieves accuracy comparable to traditional RAW-to-RGB converted images, while measuring computational efficiency gains from skipping the conversion pipeline.

Datasets

• Custom RAW/RGB Dataset — total ?; splits: test (-1)

Metrics

• top-1 classification accuracy (primary) — range: percent
  • Fraction of correctly predicted class labels out of total test samples. Results are averaged across 10 independent training runs, selecting the model with the lowest validation loss for each run.
• total computation time (s) — range: other
  • Sum of RAW-to-RGB conversion time (added only for RGB baselines) and model inference time, measured in seconds over batches of 457 samples.

Input / output format

Input: RAW sensor data (Original-RAW, Packed-RAW, or BCA-RAW) or RGB images (8-bit or 16-bit quantized), resized/padded to 40x40x1 for RAW or 40x40x3 for RGB.

Output: Single predicted class label.

Scoring recipe

correct = 0
total = 0
for img, label in test_dataset:
    pred = model(img)
    if pred == label:
        correct += 1
    total += 1
accuracy = (correct / total) * 100
return accuracy

Common pitfalls

• The reported accuracy is an average over 10 independent training runs, not a single train/val/test split.
• Total computation time for RGB baselines must include the RAW-to-RGB conversion overhead, which is not required for direct RAW classification.
• Input images are fixed to 40x40 resolution, which differs from standard image classification resolutions.

Evidence (verbatim from paper)

The test subset of the dataset is classified using each of these model parameters and the top-1 classification accuracies are measured. The average of the 10 top-1 accuracies is stored as the final top-1 classification accuracy, see Table III.

Citation

@misc{kantas2024rawinstinct,
  title={Raw Instinct: Trust Your Classifiers and Skip the Conversion},
  author={Kantas et al. (2024)},
  year={2024},
  note={arXiv:2403.14439}
}

• arXiv: 2403.14439