Iteris Merging Eval

iteris-merging-eval

IterIS: Iterative Inference-Solving Alignment for LoRA Merging — Chen et al. (2024) (arXiv:2411.15231, 2024)

What this evaluates

Evaluates the effectiveness of iterative LoRA merging (IterIS) across text-to-image diffusion, vision-language, and large language models. It probes the model's ability to preserve multiple concepts or styles without mutual interference while maintaining generation quality and task-specific performance metrics.

Datasets

• CustomConcept101 — total ?; splits: test (-1)
• DreamBooth — total ?; splits: test (-1)
• SentiCap — total ?; splits: test (-1)
• Emotion datasets (Emoint, EC, TEC, ISEAR, SUM) — total ?; splits: test (-1)
• GLUE benchmark — total ?; splits: test (-1)

Metrics

• image alignment (primary) — range: [0, 1]
  • Measures the coherence of two new concepts within the same scene, computed using CLIP-large-patch14 similarity scores averaged over 400 generated images per composition.
• text alignment — range: [0, 1]
  • Measures the consistency between textual descriptions and their corresponding visual representations, computed using CLIP-large-patch14 averaged over 400 generated images.
• style accuracy — range: percent
  • Percentage of generated captions correctly classified into the target style (positive or negative).
• CIDEr — range: [0, 1]
  • Consensus-based Image Description Evaluation metric for caption quality, averaged across test sets for both styles.
• BLEU 1-4 — range: [0, 1]
  • Precision of 1- to 4-gram overlaps between generated and reference captions, averaged across test sets.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for in-domain task integration, averaged across all task combinations.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified/generated tokens for multi-task integration, averaged across GLUE sub-tasks.
• Matthews correlation coefficient (MCC) — range: [0, 1]
  • Correlation coefficient for COLA task, linearly normalized to scale values between 0 and 1 before averaging.

Input / output format

Input: Text prompts describing 2-3 concepts (T2I), prompt-image pairs (VLM), or text inputs for classification tasks framed as generative tasks (LLM). Each task uses exactly 50 input samples for inference to derive input features.

Output: 512×512 images for T2I/VLM tasks, or generated text captions/classification outputs for LLM tasks.

Scoring recipe

def compute_metrics(predictions, gold, task_type):
    if task_type == 'T2I':
        img_align = clip_similarity(predictions, gold) # CLIP-large-patch14
        txt_align = clip_similarity(predictions, gold) # CLIP-large-patch14
        return img_align, txt_align
    elif task_type == 'VLM':
        style_acc = accuracy(predictions, gold)
        cider = cider_score(predictions, gold)
        bleu = bleu_score(predictions, gold)
        return style_acc, cider, bleu
    elif task_type == 'LLM':
        f1 = f1_score(predictions, gold)
        acc = accuracy(predictions, gold)
        mcc = mcc_score(predictions, gold)
        mcc_norm = (mcc + 1) / 2  # Linear normalization to [0,1]
        return f1, acc, mcc_norm

Common pitfalls

• Fine-tuning LoRAs with regularized data; the protocol explicitly requires using only 5-10 images without regularization.
• Evaluating on standard prompts instead of the specified 8 challenging prompts per composition pair.
• Forgetting to linearly normalize MCC to [0,1] before averaging with Accuracy in GLUE multi-task integration.

Evidence (verbatim from paper)

Following standard evaluation practices, we assess our method using two key metrics, measured by CLIP-large-patch14: (1) image alignment, which evaluates the coherence of two new concepts within the same scene, and (2) text alignment, which measures the consistency between textual descriptions and their corresponding visual representations. For each composition pair, we generate 400 images across 8 challenging prompts. Each metric is calculated as the average across 400 generated images per composition.

Citation

@misc{chen2024iteris,
  title={IterIS: Iterative Inference-Solving Alignment for LoRA Merging},
  author={Chen et al. (2024)},
  year={2024},
  note={arXiv:2411.15231}
}

• arXiv: 2411.15231