Lora Wise Eval

lora-wise-eval

Dataset Size Recovery from LoRA Weights — Salama et al. (2024) (arXiv:2406.19395, 2024)

What this evaluates

This benchmark probes a model's ability to infer the exact number of training images used to fine-tune a Low-Rank Adaptation (LoRA) adapter solely from its learned weight matrices. It evaluates how well spectral and norm-based features of LoRA parameters correlate with and reveal the scale of the underlying training dataset.

Datasets

• LoRA-WiSE — total ?; splits: train (165), test (35)

Metrics

• MAE (primary) — range: other
  • Mean Absolute Error: the average of the absolute differences between predicted and true dataset sizes. Formula: (1/N) * Σ|y_pred - y_true|.
• MAPE — range: percent
  • Mean Absolute Percentage Error: the average of the absolute percentage errors relative to the ground truth. Formula: (1/N) * Σ(|y_pred - y_true| / y_true) * 100.
• Accuracy — range: percent
  • Exact-match accuracy: the percentage of predictions that exactly equal the true dataset size.

Input / output format

Input: LoRA adapter weight matrices (specifically, the singular values or Frobenius norms of the weight matrices) from a fine-tuned Stable Diffusion model.

Output: A single integer representing the predicted number of fine-tuning images used to train the adapter.

Scoring recipe

def compute_metrics(predictions, ground_truth):
    n = len(predictions)
    mae = sum(abs(p - g) for p, g in zip(predictions, ground_truth)) / n
    mape = sum(abs(p - g) / g for p, g in zip(predictions, ground_truth)) / n * 100
    accuracy = sum(1 for p, g in zip(predictions, ground_truth) if p == g) / n * 100
    return {'MAE': mae, 'MAPE': mape, 'Accuracy': accuracy}

Common pitfalls

• Accuracy is explicitly noted as inadequate for this task because predicting a nearby size (e.g., 4 vs 5) is penalized equally to a large error, making MAPE a more appropriate metric for relative performance.
• The baseline (Frobenius-NN) and proposed method (DSiRe) are fitted separately per layer and combined via majority vote, so evaluating a single-layer prediction or ignoring the ensemble voting rule will yield incorrect results.
• Experiments are repeated 10 times with different subset samplings; reporting a single run without averaging and standard deviation over these repeats misrepresents the reported performance.

Evidence (verbatim from paper)

As described in Sec. [4.1], our main evaluation metric is Mean Absolute Error (MAE). For completeness, we choose to report two complementary metrics as well: (i) Accuracy. (ii) Mean Absolute Percentage Error (MAPE). Since DSiRe predicts dataset sizes, simple accuracy does not adequately measure its effectiveness, e.g., predicting 4 when the true value is 5 is not as bad as predicting 1. We therefore provide MAPE scores as well, which compute the percentile from the ground truth that is equal to the absolute error.

Citation

@misc{salama2024datasetsizerecovery,
  title={Dataset Size Recovery from LoRA Weights},
  author={Salama et al. (2024)},
  year={2024},
  note={arXiv:2406.19395}
}

• arXiv: 2406.19395