Ozone Eval

ozone-eval

Ozone: A Unified Platform for Transportation Research — Ou Zheng et al. (2026) (arXiv:2604.10959, 2026)

What this evaluates

Evaluates a unified platform for standardizing heterogeneous transportation trajectory data, automating cross-dataset conversion, and benchmarking safety and behavior models across multiple cities and datasets.

Datasets

• Ozone Standardized Trajectory Suite (NGSIM, highD, CitySim, UTE) — total ?; splits: train (-1), test (-1)

Metrics

• cross-city F1 score (primary) — range: [0, 1]
  • F1 score for conflict-type classification and AUC for crash-risk prediction, computed on held-out city data. Transfer efficiency is calculated as the ratio of cross-city performance to within-city performance.
• conversion error — range: meters, degrees, m/s
  • Mean absolute error between converted trajectory fields (position, heading, speed) and ground-truth annotations.
• reproducibility variance — range: percent
  • Percentage discrepancy between reproduced results and originally published numbers across standardized vs. ad-hoc preprocessing pipelines.

Input / output format

Input: Raw trajectory data (position, heading, speed, acceleration, OBB corners) from heterogeneous datasets (NGSIM, highD, CitySim, UTE) and digital-twin maps.

Output: Standardized schema fields, computed safety indicators (TTC, PET, etc.), and model predictions (conflict-type classification, crash-risk probability).

Scoring recipe

def evaluate(predictions, gold, config):
    # 1. Data conversion accuracy
    pos_err = mean(abs(predictions.pos - gold.pos))
    head_err = mean(abs(predictions.heading - gold.heading))
    speed_err = mean(abs(predictions.speed - gold.speed))
    
    # 2. Safety metrics (TTC, PET, etc.) computed via standardized pipeline
    safety_scores = compute_safety_indicators(predictions, gold)
    
    # 3. Model performance (F1/AUC)
    f1 = f1_score(gold.labels, predictions.conflict_type)
    auc = roc_auc_score(gold.labels, predictions.risk_prob)
    
    # 4. Reproducibility check
    variance = abs(reproduced_result - published_result) / published_result
    
    return {'pos_err': pos_err, 'head_err': head_err, 'speed_err': speed_err,
            'f1': f1, 'auc': auc, 'variance': variance}

Common pitfalls

• Ad-hoc preprocessing and undocumented code cause high variability in baseline results (8-22% discrepancies).
• Dataset-specific coordinate systems, formats, and naming conventions must be explicitly normalized before cross-city evaluation.
• Safety metric thresholds (e.g., TTC, PET) must be consistently defined across datasets to ensure comparability.

Evidence (verbatim from paper)

The average cross-city F1 score for conflict-type classification was 0.83, compared with 0.91 for within-city evaluation, corresponding to a transfer efficiency of 91%. For crash-risk prediction, the cross-city AUC was 0.79, compared with 0.86 for within-city evaluation (92% transfer efficiency).

Citation

@misc{zheng2026ozone,
  title={Ozone: A Unified Platform for Transportation Research},
  author={Ou Zheng et al. (2026)},
  year={2026},
  note={arXiv:2604.10959}
}

• arXiv: 2604.10959