Easytpp Eval

easytpp-eval

EasyTPP: Towards Open Benchmarking Temporal Point Processes — Xue et al. (2023) (arXiv:2307.08097, 2023)

What this evaluates

Evaluates neural Temporal Point Process models on event sequence prediction tasks, specifically forecasting the timing and categorical type of future events given historical sequences.

Datasets

• Retweet — total ?; splits: test (-1)
• Taxi — total ?; splits: test (-1)

Metrics

• TIME RMSE (primary) — range: other
  • Root Mean Squared Error between predicted and actual event timestamps. Lower values indicate better temporal prediction accuracy.
• TYPE ERROR RATE — range: percent
  • Fraction of incorrectly predicted event types compared to ground truth. Lower values indicate better categorical prediction accuracy.

Input / output format

Input: Historical sequence of event timestamps and event types.

Output: Predicted timestamp and event type for the next event.

Scoring recipe

def compute_metrics(pred_times, gold_times, pred_types, gold_types):
    time_rmse = math.sqrt(mean((p - g)**2 for p, g in zip(pred_times, gold_times)))
    type_err = sum(1 for p, g in zip(pred_types, gold_types) if p != g) / len(gold_types)
    return {'TIME RMSE': time_rmse, 'TYPE ERROR RATE': type_err}

Common pitfalls

• Early stopping is performed on the held-out dev set using log-likelihood, not the test set.
• Results are averaged over 5 independent runs to account for randomness.
• Models are evaluated on fixed horizon splits (e.g., AVG 5 EVENTS, AVG 10 EVENTS) rather than variable lengths.

Evidence (verbatim from paper)

METRICS (TIME RMSE / TYPE ERROR RATE)

Citation

@misc{xue2023easytpp,
  title={EasyTPP: Towards Open Benchmarking Temporal Point Processes},
  author={Xue et al. (2023)},
  year={2023},
  note={arXiv:2307.08097}
}

• arXiv: 2307.08097