Trajectory Generation Eval

trajectory-generation-eval

Revisiting Synthetic Human Trajectories: Imitative Generation and Benchmarks Beyond Datasaurus — Deng et al. (2024) (arXiv:2409.13790, 2024)

What this evaluates

Evaluates the statistical fidelity and practical utility of synthetic human trajectory generation models by measuring how well generated trajectories perform on downstream mobility tasks compared to real trajectories. It probes whether synthetic data can replace real data without performance degradation across recommendation, prediction, labeling, and simulation tasks.

Datasets

• Foursquare Tokyo (TKY) — total ?; splits: (unstated); repo https://github.com/UM-Data-Intelligence-Lab/MIRAGE
• Foursquare Istanbul (IST) — total ?; splits: (unstated); repo https://github.com/UM-Data-Intelligence-Lab/MIRAGE
• Foursquare New York City (NYC) — total ?; splits: (unstated); repo https://github.com/UM-Data-Intelligence-Lab/MIRAGE

Metrics

• MAPE (primary) — range: percent
  • Mean Absolute Percentage Error. Computes the average absolute percentage difference between task performance metrics (e.g., MRR, NDCG, Accuracy) obtained on real trajectories versus generated trajectories. Lower values indicate better utility fidelity.
• MSPE — range: percent
  • Mean Squared Percentage Error. Computes the average squared percentage difference between real and generated trajectory task performance. Lower values indicate better utility fidelity.
• JSD — range: [0, 1]
  • Jensen-Shannon Divergence. Measures the statistical similarity between the distributions of real and generated trajectories across five aspects: Distance, Radius of gyration, Interval, DailyLoc, and Category. Lower values indicate better distributional fidelity.

Input / output format

Input: Real user trajectory datasets containing sequences of (user, location, timestamp) pairs collected from location-based social networks.

Output: Synthetic trajectory datasets with the exact same number of trajectories as the real datasets, generated without exposing any real trajectories.

Scoring recipe

def evaluate(real_trajectories, synthetic_trajectories, task_algorithms, task_metrics):
    # Run downstream tasks on both real and synthetic data
    real_perf = [run_task(real_trajectories, algo, metric) for algo in task_algorithms for metric in task_metrics]
    synth_perf = [run_task(synthetic_trajectories, algo, metric) for algo in task_algorithms for metric in task_metrics]
    
    # Compute paired performance discrepancy
    mape = mean(abs(r - s) / r for r, s in zip(real_perf, synth_perf))
    mspe = mean(((r - s) / r) ** 2 for r, s in zip(real_perf, synth_perf))
    return mape, mspe

Common pitfalls

• Data leakage: Augmenting real data with synthetic data instead of evaluating strictly on synthetic-only data inflates utility scores.
• Single-technique bias: Using only one downstream algorithm or metric skews the utility evaluation; results must be averaged across multiple algorithms and metrics.
• Ignoring statistical fidelity: Focusing solely on task performance without measuring distributional similarity (e.g., JSD) misses core generation quality issues.

Evidence (verbatim from paper)

We then measure the paired performance discrepancy between the real and generated trajectories using Mean Absolute Percentage Error (MAPE) and Mean Squared Percentage Error (MSPE), which serve as final benchmarks to assess the ultimate utility of the generated trajectories.

Citation

@misc{deng2024revisiting,
  title={Revisiting Synthetic Human Trajectories: Imitative Generation and Benchmarks Beyond Datasaurus},
  author={Deng et al. (2024)},
  year={2024},
  note={arXiv:2409.13790}
}

• arXiv: 2409.13790