Cond P Diff Eval

cond-p-diff-eval

Conditional LoRA Parameter Generation — Jin et al. (2024) (arXiv:2408.01415, 2024)

What this evaluates

Evaluates a conditional latent diffusion framework's ability to synthesize task-specific LoRA parameters for NLP and image style-transfer tasks. It probes whether generated parameters can match or exceed standard fine-tuning and model-averaging baselines across diverse domains.

Datasets

• GLUE benchmark — total ?; splits: validation (-1)
• SemArt — total ?; splits: test (-1)
• WikiArt — total ?; splits: test (-1)

Metrics

• Average accuracy (primary) — range: percent
  • Arithmetic mean of classification accuracies across six GLUE tasks: SST2, RTE, MRPC, COLA, QNLI, and STSB.
• FID — range: other
  • Fréchet Inception Distance measuring the distance between feature distributions of real and generated images. Lower scores indicate better image generation quality.

Input / output format

Input: Task description and few-shot examples (NLP) or style images (CV) condition a diffusion model to generate LoRA parameters, which are merged into pre-trained models for inference.

Output: Per-task accuracy scores and their average for GLUE; FID scores for style-transfer image generation.

Scoring recipe

def compute_glue_avg(acc_sst2, acc_rte, acc_mrpc, acc_cola, acc_qnli, acc_stsb):
    return (acc_sst2 + acc_rte + acc_mrpc + acc_cola + acc_qnli + acc_stsb) / 6

def compute_fid(real_images, generated_images):
    # Compute FID between real and generated image feature distributions
    return frechet_inception_distance(real_images, generated_images)

Common pitfalls

• FID is an inverted metric: lower values indicate better performance.
• GLUE evaluation only averages six specific tasks (SST2, RTE, MRPC, COLA, QNLI, STSB), not the full benchmark.
• Model selection for NLP is based on the best performance on the training dataset, not the validation set, before reporting validation metrics.
• Baselines include 'original' (best validation performance of standard LoRA fine-tuning) and 'model soup' (averaged weights of multiple fine-tuned models).

Evidence (verbatim from paper)

In NLP experiments, we test on the language understanding GLUE benchmark [[53]]. In CV experiments, we focus on the style-transfer tasks. We use the SemArt and WikiArt datasets [[10], [45]], which contain diverse artistic images, and evaluate them using the Fréchet Inception Distance (FID, [[15]], as employed by StyleGAN [[23]], with lower scores indicating better performance.

Citation

@misc{jin2024conditionallora,
  title={Conditional LoRA Parameter Generation},
  author={Jin et al. (2024)},
  year={2024},
  note={arXiv:2408.01415}
}

• arXiv: 2408.01415