Cesped Pose Estimation Eval

cesped-pose-estimation-eval

CESPED: a new benchmark for supervised particle pose estimation in Cryo-EM — Sanchez-Garcia et al. (2023) (arXiv:2311.06194, 2023)

What this evaluates

This benchmark evaluates supervised deep learning models for predicting 3D particle orientations (rotation matrices) from 2D Cryo-EM micrographs. It assesses both angular prediction accuracy and the downstream quality of 3D structural reconstructions derived from the predicted poses.

Datasets

• CESPED — total ?; splits: half_0 (-1), half_1 (-1); repo https://github.com/rsanchezgarc/cesped

Metrics

• MAnE (primary) — range: degrees
  • Mean Angular Error: average of the minimum angular distance between predicted and ground-truth rotation matrices over a symmetry group G. Formula: MAnE = (1/N) ∑ min_{g_j∈G} arccos((trace(g_j·trueR_i·predR_i^T)-1)/2).
• wMAnE — range: degrees
  • Confidence-weighted mean angular error. Weights each angular error by the ground-truth pose confidence (Relion's rlnMaxValueProbDistribution). Used for hyperparameter tuning.
• PCC — range: [0, 1]
  • Real space Pearson’s Correlation Coefficient between ground-truth and predicted 3D volumes. Measures voxel-wise linear correlation.
• FSCR_0.143 — range: Å (resolution)
  • Fourier Shell Correlation Resolution at threshold 0.143. The highest frequency shell where FSC drops below 0.143, indicating resolution where maps agree with SNR=0.5.

Input / output format

Input: 2D Cryo-EM particle images (downsampled to 1.5 Å/pixel, background-normalized to mean 0/std 1, phase-flipped for defocus, and cropped to exclude neighboring particles).

Output: Probability distribution over a discretised grid of rotation matrices (SO(3)_grid), optionally expanded for point symmetry groups.

Scoring recipe

# Calculate MAnE
ang_errors = []
for true_R, pred_R in zip(true_rotations, pred_rotations):
    min_err = min(arccos((trace(g * true_R * pred_R.T) - 1) / 2) for g in symmetry_group)
    ang_errors.append(min_err)
mane = sum(ang_errors) / len(ang_errors)

# Calculate PCC
pcc = pearsonr(volume_gt.flatten(), volume_pred.flatten())

# Calculate FSCR_0.143
fsc_curve = fourier_shell_correlation(volume_gt, volume_pred)
fscr = find_first_index_below_threshold(fsc_curve, 0.143)

Common pitfalls

• Using ground-truth translations during volume reconstruction overestimates performance, as translation accuracy is tightly coupled with angular estimation.
• Symmetry expansion in labels must be handled during evaluation; failing to account for the symmetry group G will incorrectly penalize valid symmetric predictions.
• Ground-truth pose uncertainty is inherent in Cryo-EM; standard MAnE does not account for this, which is why wMAnE and ΔPCC/ΔFSC are recommended for robust benchmarking.

Evidence (verbatim from paper)

The most employed metric in pose estimation is the mean angular error, MAnE=∑ angError_i where angError_i=min_{g_j∈G} arccos((trace(g_j·trueR_i·predR_i^T)-1)/2) with G the set of rotation matrices given a point symmetry group, and trueR_i and predR_i the ground-truth and predicted rotation matrices. The quality of the volumes reconstructed from the predicted poses is assessed by comparing them against the ground-truth volumes generated from the original poses. As comparison metrics we use the real space Pearson’s Correlation Coefficient, PCC(X,Y)=... and the Fourier Shell Correlation Resolution at threshold t, FSCR_t(X,Y)...

Citation

@misc{sanchezgarcia2023cesped,
  title={CESPED: a new benchmark for supervised particle pose estimation in Cryo-EM},
  author={Sanchez-Garcia et al. (2023)},
  year={2023},
  note={arXiv:2311.06194}
}

• arXiv: 2311.06194