Utility Aware Data Pricing Eval

utility-aware-data-pricing-eval

Utility-Aware Data Pricing: Token-Level Quality and Empirical Training Gain for LLMs — Minghui Xu, Qi Luo, and Kun Li (2026) (arXiv:2604.22893, 2026)

What this evaluates

Evaluates whether token-level quality signals and empirical training gain metrics can accurately predict real data utility for LLM fine-tuning, outperforming traditional row- or token-count baselines. It probes the framework's predictive alignment, ranking fidelity, and robustness to adversarial or low-value data across multiple domains.

Datasets

• Alpaca — total ?; splits: train (-1), val (-1)
• GSM8K — total ?; splits: train (-1), val (-1)
• CodeXGLUE-Python — total ?; splits: train (-1), val (-1)

Metrics

• Spearman rank correlation (primary) — range: [-1, 1]
  • Measures the monotonic relationship between predicted valuation scores and realized model improvement gains. Calculated as the rank correlation coefficient between the two vectors.
• Top-k retrieval accuracy — range: [0, 1]
  • Tests whether the predicted top-k sources match the true top-k contributors based on realized gain. Computed as the fraction of correct top-k matches.
• Kendall rank correlation — range: [-1, 1]
  • Measures ranking stability between predicted scores and realized gains by counting concordant and discordant pairs.
• Mean absolute error (z-score normalized) — range: [0, ∞)
  • Quantifies calibration quality by computing the mean absolute difference between z-score normalized predicted scores and realized gains.

Input / output format

Input: Source-level data partitions $D_i$ along with token-level quality scores, proxy model training gains, influence approximations, and Data Shapley values.

Output: Unified valuation score per source, ranked list of sources, and evaluation metrics (correlation, accuracy, error).

Scoring recipe

def evaluate_valuation(predicted_scores, realized_gains, k=2):
    # Realized gain: G_i = V_target(D) - V_target(D \ {D_i})
    # Spearman rank correlation
    rho = spearmanr(predicted_scores, realized_gains).correlation
    # Top-k retrieval accuracy
    pred_top_k = np.argsort(predicted_scores)[-k:][::-1]
    true_top_k = np.argsort(realized_gains)[-k:][::-1]
    topk_acc = np.mean([np.array_equal(pred_top_k, true_top_k)])
    # MAE after z-score normalization
    z_pred = (predicted_scores - np.mean(predicted_scores)) / np.std(predicted_scores)
    z_true = (realized_gains - np.mean(realized_gains)) / np.std(realized_gains)
    mae = np.mean(np.abs(z_pred - z_true))
    return rho, topk_acc, mae

Common pitfalls

• Relying on quantity-based baselines (row-count, token-count) which are fundamentally misaligned with actual training benefit.
• Using lightweight Hessian approximations for influence functions at small sample scales, which can yield strongly negative correlations.
• Over-interpreting smoke-test results with very small sample sizes (12 train, 6 val) as final benchmark performance.

Evidence (verbatim from paper)

We compare each predicted valuation score against $G_{i}$ using four primary metrics: Spearman rank correlation to capture the monotonic relationship between predicted scores and realized gains; Kendall rank correlation for ranking stability; Top-$k$ retrieval accuracy to test whether the predicted top-$k$ sources match the true top-$k$ contributors; and mean absolute error after z-score normalization to quantify calibration quality.

Citation

@misc{xu2026utilityaware,
  title={Utility-Aware Data Pricing: Token-Level Quality and Empirical Training Gain for LLMs},
  author={Minghui Xu, Qi Luo, and Kun Li (2026)},
  year={2026},
  note={arXiv:2604.22893}
}

• arXiv: 2604.22893