Sensor Invariant Tactile Eval

sensor-invariant-tactile-eval

Sensor-Invariant Tactile Representation — Gupta et al. (2025) (arXiv:2502.19638, 2025)

What this evaluates

Evaluates the zero-shot transferability of tactile representations across different physical sensors for shape reconstruction, object classification, and 3D pose estimation. It probes whether learned features generalize across varying optical designs and manufacturing differences without retraining.

Datasets

• Real-world tactile contact dataset — total ?; splits: train (-1), test (-1)

Metrics

• top-1 accuracy (primary) — range: percent
  • Percentage of correctly classified objects out of the total 16 classes. Computed as (number of correct predictions / total samples) * 100.
• RMSE (mm) — range: other
  • Root Mean Square Error of the predicted 3-DoF (x,y,z) position change compared to ground truth, measured in millimeters.

Input / output format

Input: Tactile contact images captured by GelSight Mini, Wedge, Hex, or DIGIT sensors. For pose estimation, pairs of initial and final tactile images of the same object.

Output: Classification: discrete object label (1 of 16 classes). Pose estimation: 3-DoF (x,y,z) position change in mm. Shape reconstruction: 3D height/normal map (qualitative).

Scoring recipe

def compute_transfer_metric(sensor_set, predictions, ground_truth):
    n = len(sensor_set)
    cross_sensor_scores = []
    for i, train_sensor in enumerate(sensor_set):
        for j, test_sensor in enumerate(sensor_set):
            if i != j:
                score = evaluate(predictions[i][j], ground_truth[i][j])
                cross_sensor_scores.append(score)
    return sum(cross_sensor_scores) / (n * (n - 1))

def compute_accuracy(preds, gold):
    return (preds == gold).mean() * 100

def compute_rmse(preds, gold):
    return np.sqrt(np.mean((preds - gold) ** 2))

Common pitfalls

• Confusing intra-sensor transfer (same optical design, different coatings) with inter-sensor transfer (different optical structures), which have vastly different difficulty levels.
• Treating the 'no-transfer' (same-sensor train/test) score as a transfer metric; it actually serves as an upper bound for performance.
• Assuming ImageNet pre-training improves tactile regression tasks; the paper shows it only marginally helps pose estimation.

Evidence (verbatim from paper)

We evaluate model performance on three downstream tasks: shape reconstruction, object classification, and contact localization. ... The transfer performance across all sensors in the set is computed as Transfer Performance = 1/(n(n-1)) sum_{i=1}^n sum_{j!=i}^n A_{ij} ... We compare SITR with baselines using our real-world classification dataset from Section 4.2 and report top-1 accuracy.

Citation

@misc{gupta2025sensorinvariant,
  title={Sensor-Invariant Tactile Representation},
  author={Gupta et al. (2025)},
  year={2025},
  note={arXiv:2502.19638}
}

• arXiv: 2502.19638