Temporal Domain Generalization Eval

temporal-domain-generalization-eval

Scaling Up Temporal Domain Generalization via Temporal Experts Averaging — Aoming Liu et al. (arXiv:2509.26045, 2025)

What this evaluates

Evaluates a model's ability to generalize to future, unseen temporal domains without full retraining. It measures out-of-distribution accuracy on sequentially arriving target domains after training on historical source domains.

Datasets

• Yearbook — total ?; splits: source_train (-1), source_val (-1), target_test (-1)
• Rotated MNIST (RMNIST) — total ?; splits: source_train (-1), source_val (-1), target_test (-1)
• FMoW — total ?; splits: source_train (-1), source_val (-1), target_test (-1)
• Huffpost — total ?; splits: source_train (-1), source_val (-1), target_test (-1)
• Arxiv — total ?; splits: source_train (-1), source_val (-1), target_test (-1)
• CLEAR-10/100 — total ?; splits: source_train (-1), source_val (-1), target_test (-1)

Metrics

• OOD_avg accuracy (primary) — range: percent
  • Percentage of correctly predicted class labels on the target domain(s). Computed as the average accuracy across all target domains.

Input / output format

Input: Image or text data samples from sequentially ordered source domains, with the model required to predict class labels for sequentially ordered target domains.

Output: Class predictions (or probabilities) for each target domain instance.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

Common pitfalls

• Confusing Temporal Domain Generalization (TDG) with standard Domain Generalization (DG) or Continual Domain Generalization (CDGTD), which have different data access and training constraints.
• Failing to respect the chronological domain ordering and specified source-to-target ratios (e.g., 16:5 for Yearbook), which breaks the temporal shift assumption.
• Overlooking computational constraints that force some baselines (e.g., GI, DRAIN) to skip finetuning or use classifier-only variants on larger benchmarks, skewing direct performance comparisons.

Evidence (verbatim from paper)

Each dataset is divided into first S source and last F target domains with ratios S:F of: Yearbook (16:5), RMNIST (6:3), FMoW (13:3), Huffpost (4:3), Arxiv (9:7), and CLEAR-10/100 (5:5). Each source domain uses a random 90%-10% train-validation split. We report OOD_avg accuracy (%) across different buffer sizes (1%, 3%, 5%, 10%, 20).

Citation

@misc{liu2025tea,
  title={Scaling Up Temporal Domain Generalization via Temporal Experts Averaging},
  author={Aoming Liu et al.},
  year={2025},
  note={arXiv:2509.26045}
}

• arXiv: 2509.26045