Influence Estimation Eval

influence-estimation-eval

DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models — Kwon et al. (2023) (arXiv:2310.00902, 2023)

What this evaluates

This protocol evaluates the accuracy and efficiency of influence function approximation methods (DataInf, LiSSA, Hessian-free) in matching exact influence values, detecting mislabeled training data, and identifying training points that most impact a test instance's loss across text and image generation tasks.

Datasets

• GLUE (binary classification subsets) — total ?; splits: train (-1)
• Custom Text Generation Datasets — total 300; splits: train (270), test (30)
• Custom Image Generation Datasets — total 600; splits: train (600), test (150)

Metrics

• AUC (primary) — range: [0, 1]
  • Area under the receiver operating characteristic curve. Measures the probability that a score randomly selected from a positive class (e.g., mislabeled data or same-class training points) is greater than that of a negative class.
• Pearson correlation coefficient — range: [-1, 1]
  • Linear correlation between the estimated influence values and the exact influence function values. Higher values indicate better approximation accuracy.
• Recall — range: [0, 1]
  • Percentage of training points with the same class as the test example among the s smallest influential training points, where s equals the number of training examples per class.

Input / output format

Input: Fine-tuned model parameters (LoRA), a training dataset, and a single test instance (or test dataset).

Output: A scalar influence value for each training data point, representing its estimated impact on the test instance's loss.

Scoring recipe

def compute_auc(influence_scores, labels):
    return roc_auc_score(labels, influence_scores)

def compute_correlation(exact_inf, approx_inf):
    return pearsonr(exact_inf, approx_inf)[0]

def compute_recall(influence_scores, labels, s):
    top_s_idx = argsort(influence_scores)[:s]
    return mean(labels[top_s_idx])

Common pitfalls

• LiSSA frequently fails to converge on high-dimensional models due to iterative updates, producing unstable or invalid influence values.
• Exact influence function values can be artificially high for ambiguous but correctly labeled data near decision boundaries, making it suboptimal for mislabel detection compared to approximations.
• The damping parameter lambda_l significantly impacts performance; improper scaling per layer/dimension can degrade both exact and approximate methods.

Evidence (verbatim from paper)

As for the evaluation metric, we use the area under the curve (AUC) score between influence values and the binary annotations for mislabeled data to capture the quality of the influence function values. This AUC measures the probability that a score randomly selected from a class of mislabeled data is greater than that of a class of clean data.

Citation

@misc{kwon2023datainf,
  title={DataInf: Efficiently Estimating Data Influence in LoRA-tuned LLMs and Diffusion Models},
  author={Kwon et al. (2023)},
  year={2023},
  note={arXiv:2310.00902}
}

• arXiv: 2310.00902