Next Transaction Prediction Eval

next-transaction-prediction-eval

PANTHER: Generative Pretraining Beyond Language for Sequential User Behavior Modeling — Li et al. (2025) (arXiv:2510.10102, 2025)

What this evaluates

Evaluates a model's ability to predict the next user transaction or item interaction based on historical sequential behavior. It probes the model's capacity to capture periodic patterns, user-specific preferences, and generalize across financial and recommendation domains.

Datasets

• WeChat Pay — total 5300000000; splits: train (-1), val (-1), test (-1); repo https://github.com/yzhangjy/PANTHER
• CCT — total 20000000; splits: train (-1), val (-1), test (-1)
• MBD-mini — total ?; splits: train (-1), val (-1), test (-1)
• MovieLens-1M — total 1000000; splits: train (-1), val (-1), test (-1); HF ml-1m
• Yelp — total ?; splits: train (-1), val (-1), test (-1); HF yelp

Metrics

• HR@1 (primary) — range: [0, 1]
  • Hit Ratio at 1: 1 if the ground-truth item is ranked #1 in the predicted list, else 0. Averaged over test instances.
• HR@5 — range: [0, 1]
  • Hit Ratio at 5: 1 if the ground-truth item appears in the top-5 predicted items, else 0. Averaged over test instances.
• HR@10 — range: [0, 1]
  • Hit Ratio at 10: 1 if the ground-truth item appears in the top-10 predicted items, else 0. Averaged over test instances.
• NDCG@5 — range: [0, 1]
  • Normalized Discounted Cumulative Gain at 5: Ranks predicted items by score, discounts gains logarithmically by position, and normalizes by the ideal DCG. Averaged over test instances.

Input / output format

Input: Tokenized sequence of historical user transactions (attributes: amount, merchant category, etc.)

Output: Ranked list of candidate next transactions/items, or top-K predictions.

Scoring recipe

def compute_hr_at_k(preds, gold, k):
    return 1.0 if gold in preds[:k] else 0.0

def compute_ndcg_at_k(preds, gold, k):
    dcg = 0.0
    for i, item in enumerate(preds[:k]):
        if item == gold:
            dcg = 1.0 / math.log2(i + 2)
            break
    idcg = 1.0 / math.log2(2)
    return dcg / idcg if idcg > 0 else 0.0

# Aggregate over test set
hr_scores = [compute_hr_at_k(preds, gold, k) for preds, gold in test_data]
ndcg_scores = [compute_ndcg_at_k(preds, gold, k) for preds, gold in test_data]
return sum(hr_scores) / len(hr_scores), sum(ndcg_scores) / len(ndcg_scores)

Common pitfalls

• Must use full-ranking evaluation (~3,700 items) rather than sampled negatives to match the paper's protocol.
• Splits are chronological, not random, to prevent data leakage from future transactions.

Evidence (verbatim from paper)

Specifically, HR@K measures the fraction of test instances in which the ground-truth item appears among the top-K predicted items. NDCG@K assesses the ranking quality by assigning higher weights to relevant items placed at top positions, normalized by the ideal discounted gain.

Citation

@misc{li2025panther,
  title={PANTHER: Generative Pretraining Beyond Language for Sequential User Behavior Modeling},
  author={Li et al. (2025)},
  year={2025},
  note={arXiv:2510.10102}
}

• arXiv: 2510.10102