gess-ood-eval
GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shifts — Zou et al. (2023) (arXiv:2310.08677, 2023)
What this evaluates
Evaluates geometric deep learning models' out-of-distribution generalization across scientific domains under conditional, covariate, and concept shifts. It probes how different learning paradigms (ERM, domain adaptation, transfer learning, and OOD generalization) perform when provided with varying amounts of target-domain data.
Datasets
- Track (Particle Tracking Simulation) — total ?; splits: Train-ID (-1), Val-ID (-1), Test-ID (-1), Train-OOD (-1), Val-OOD (-1), Test-OOD (-1)
- QMOF (Quantum Metal-organic Frameworks) — total ?; splits: Train-ID (-1), Val-ID (-1), Test-ID (-1), Train-OOD (-1), Val-OOD (-1), Test-OOD (-1)
- DrugOOD-3D (3D Conformers of Drug Molecules) — total ?; splits: Train-ID (-1), Val-ID (-1), Test-ID (-1), Train-OOD (-1), Val-OOD (-1), Test-OOD (-1)
Metrics
MAE (primary) — range: other
- Mean Absolute Error, calculated as the average of the absolute differences between predicted and true values across all test instances.
Input / output format
Input: Geometric or graph-structured data representing scientific objects (e.g., particle collision events, molecular frameworks, drug conformers) with associated node/edge features and target labels.
Output: Predicted continuous values (regression) or class labels corresponding to the input geometric structures.
Scoring recipe
def compute_mae(predictions, gold_labels):
errors = [abs(p - g) for p, g in zip(predictions, gold_labels)]
return sum(errors) / len(errors)
Common pitfalls
- Assuming OOD generalization methods (trained only on ID data) will consistently outperform ERM; the paper shows they often fail to provide significant improvements in practical scientific scenarios.
- Assuming fine-tuning with few OOD labels always improves performance; it can cause catastrophic forgetting or underperform when the distribution shift is mild or labeled data is too scarce.
- Assuming subgroup splits for methods like GroupDRO accurately reflect the true distribution shift; subjective splits may misalign with the actual shift mechanism, degrading robustness.
Evidence (verbatim from paper)
We find this strategy performs particularly well in the case of HSE06 (even fine-tuned on a very small number of OOD labels), where it largely outperforms all other methods with the MAE score increased by at least 30%.
Citation
@misc{zou2023gess,
title={GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shifts},
author={Zou et al. (2023)},
year={2023},
note={arXiv:2310.08677}
}
1---2name: gess-ood-eval3description: Evaluates geometric deep learning models' out-of-distribution generalization across scientific domains under conditional, covariate, and concept shifts. It probes how different learning paradigms (ERM, domain adaptation, transfer learning, and OOD generalization) perform when provided with varying amounts of target-domain data. Use when the user wants to benchmark on Track (Particle Tracking Simulation), QMOF (Quantum Metal-organic Frameworks), DrugOOD-3D (3D Conformers of Drug Molecules), or asks about evaluating this task. Reports MAE.4---56# gess-ood-eval78> GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shifts — Zou et al. (2023) (arXiv:2310.08677, 2023)910## What this evaluates1112Evaluates geometric deep learning models' out-of-distribution generalization across scientific domains under conditional, covariate, and concept shifts. It probes how different learning paradigms (ERM, domain adaptation, transfer learning, and OOD generalization) perform when provided with varying amounts of target-domain data.1314## Datasets1516- **Track (Particle Tracking Simulation)** — total ?; splits: Train-ID (-1), Val-ID (-1), Test-ID (-1), Train-OOD (-1), Val-OOD (-1), Test-OOD (-1)17- **QMOF (Quantum Metal-organic Frameworks)** — total ?; splits: Train-ID (-1), Val-ID (-1), Test-ID (-1), Train-OOD (-1), Val-OOD (-1), Test-OOD (-1)18- **DrugOOD-3D (3D Conformers of Drug Molecules)** — total ?; splits: Train-ID (-1), Val-ID (-1), Test-ID (-1), Train-OOD (-1), Val-OOD (-1), Test-OOD (-1)1920## Metrics2122- `MAE` **(primary)** — range: other23 - Mean Absolute Error, calculated as the average of the absolute differences between predicted and true values across all test instances.2425## Input / output format2627**Input**: Geometric or graph-structured data representing scientific objects (e.g., particle collision events, molecular frameworks, drug conformers) with associated node/edge features and target labels.2829**Output**: Predicted continuous values (regression) or class labels corresponding to the input geometric structures.3031## Scoring recipe3233```python34def compute_mae(predictions, gold_labels):35 errors = [abs(p - g) for p, g in zip(predictions, gold_labels)]36 return sum(errors) / len(errors)37```3839## Common pitfalls4041- Assuming OOD generalization methods (trained only on ID data) will consistently outperform ERM; the paper shows they often fail to provide significant improvements in practical scientific scenarios.42- Assuming fine-tuning with few OOD labels always improves performance; it can cause catastrophic forgetting or underperform when the distribution shift is mild or labeled data is too scarce.43- Assuming subgroup splits for methods like GroupDRO accurately reflect the true distribution shift; subjective splits may misalign with the actual shift mechanism, degrading robustness.4445## Evidence (verbatim from paper)4647> We find this strategy performs particularly well in the case of HSE06 (even fine-tuned on a very small number of OOD labels), where it largely outperforms all other methods with the MAE score increased by at least 30%.4849## Citation5051```bibtex52@misc{zou2023gess,53 title={GeSS: Benchmarking Geometric Deep Learning under Scientific Applications with Distribution Shifts},54 author={Zou et al. (2023)},55 year={2023},56 note={arXiv:2310.08677}57}58```5960- arXiv: 2310.08677
qhjqhj00