Foura Eval

foura-eval

FouRA: Fourier Low Rank Adaptation — Borse et al. (2024) (arXiv:2406.08798, 2024)

What this evaluates

Evaluates the capability of a Fourier-domain low-rank adapter to generate high-quality, diverse images for style transfer and concept editing. It also assesses the adapter's performance on standard language understanding benchmarks compared to baseline adapters like LoRA.

Datasets

• Paintings — total 630; splits: eval (630)
• Blue-Fire — total 900; splits: eval (900)
• 3D — total ?; splits: eval (-1)
• Origami — total ?; splits: eval (-1)
• GLUE — total ?; splits: test (-1)

Metrics

• HPSv2.1 (primary) — range: other
  • Evaluates the measure of image quality and alignment with the prompt/style using a pretrained vision-language model.
• LPIPS diversity — range: [0, 1]
  • Captures the diversity within all possible pairs of generated images across random seeds by computing pairwise LPIPS distances.
• LPIPS similarity — range: [0, 1]
  • Measures perceptual similarity between edited images and the original base image using LPIPS distance.
• GLUE score — range: percent
  • Standard accuracy or F1 score averaged across MNLI, CoLA, SST2, STSB, MRPC, and QNLI tasks.

Input / output format

Input: Text prompts describing styles or editing concepts for image generation; sentence pairs or single sentences for GLUE classification/regression tasks.

Output: Generated images for vision tasks; classification labels or regression scores for GLUE tasks.

Scoring recipe

# Vision: HPSv2.1 & LPIPS Diversity
hps_scores = [hpsv2_model(prompt, img) for img in generated_images]
hps_avg = mean(hps_scores)

# Vision: LPIPS Diversity
distances = [lpips_distance(img_i, img_j) for i, img_i in enumerate(images) for j, img_j in enumerate(images) if i < j]
diversity_score = mean(distances)

# Language: GLUE
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = correct / len(gold_labels)

Common pitfalls

• High adapter strength ($\alpha$) amplifies visual artifacts in LoRA but FouRA remains stable, so comparing at fixed $\alpha$ without checking for distortion is misleading.
• Rank is highly sensitive for LoRA, requiring careful tuning, whereas FouRA maintains robust performance across varying ranks due to frequency-domain gating.
• Results must be averaged over multiple random seeds (30 for vision, 3 for language) to account for stochastic generation and adapter initialization.

Evidence (verbatim from paper)

Metrics: For quantifying the quality of images generated by FouRA and LoRA finetuned diffusion models, we report HPSv2.1*[[39]]* and LPIPS diversity [[44]] scores. The HPSv2 metric evaluates the measure of the image quality, and alignment with the prompt/style. LPIPS diversity score captures the diversity within all possible pairs of generated images across seeds. We provide an in-depth analysis of these metrics in Appendix[D]. For the image editing task, we compare edited images using LPIPS similarity (compared to the base image). For language models, we report on the General Language Understanding Evaluation (GLUE) benchmarks*[[36]]*, see details in App.[C.1].

Citation

@misc{borse2024foura,
  title={FouRA: Fourier Low Rank Adaptation},
  author={Borse et al. (2024)},
  year={2024},
  note={arXiv:2406.08798}
}

• arXiv: 2406.08798