Knowcusbench Eval

knowcusbench-eval

MoKus: Leveraging Cross-Modal Knowledge Transfer for Knowledge-Aware Concept Customization — Zhu et al. (2026) (arXiv:2603.12743, 2026)

What this evaluates

Evaluates a model's ability to bind natural language knowledge to visual concepts for high-fidelity image reconstruction and customized generation without full retraining. It probes cross-modal knowledge transfer, concept fidelity, and prompt alignment in diffusion-based image generation.

Datasets

• KnowCusBench — total ?; splits: test (-1)

Metrics

• CLIP-I — range: [0, 1]
  • Calculates the average pairwise cosine similarity between CLIP embeddings of the generated images and the real reference images.
• CLIP-I-Seg (primary) — range: [0, 1]
  • Employs SAM3 to segment the target concept from generated images, then calculates the average pairwise cosine similarity between CLIP embeddings of the segmented generation and the real images to isolate concept fidelity from background noise.
• CLIP-T — range: [0, 1]
  • Computes the average cosine similarity between text prompt embeddings and image embeddings to measure prompt fidelity.
• Pick Score — range: [0, 1]
  • Evaluates human preference by measuring the alignment between text and image embeddings using a pretrained PickScore model.

Input / output format

Input: Text prompts specifying the target concept and associated knowledge, optionally paired with reference images for the target concept.

Output: Generated images conditioned on the text prompts.

Scoring recipe

def compute_metrics(predictions, gold_images, prompts):
    # predictions: list of generated images
    # gold_images: list of reference images
    # prompts: list of text prompts
    
    clip_i_scores = []
    clip_i_seg_scores = []
    clip_t_scores = []
    pick_scores = []
    
    for pred_img, ref_img, prompt in zip(predictions, gold_images, prompts):
        # CLIP-I
        clip_i_scores.append(cosine_similarity(clip_embed(pred_img), clip_embed(ref_img)))
        
        # CLIP-I-Seg
        seg_img = sam3_segment(pred_img, target_concept)
        clip_i_seg_scores.append(cosine_similarity(clip_embed(seg_img), clip_embed(ref_img)))
        
        # CLIP-T
        clip_t_scores.append(cosine_similarity(clip_embed(prompt), clip_embed(pred_img)))
        
        # Pick Score
        pick_scores.append(pickscore_model(prompt, pred_img))
        
    return {
        'CLIP-I': mean(clip_i_scores),
        'CLIP-I-Seg': mean(clip_i_seg_scores),
        'CLIP-T': mean(clip_t_scores),
        'Pick Score': mean(pick_scores)
    }

Common pitfalls

• Background elements in generated images can artificially inflate fidelity scores if not segmented, making CLIP-I-Seg more reliable than raw CLIP-I.
• Finetuning the LLM encoder directly can severely alter the latent space distribution, causing poor prompt-image alignment and degraded generation quality.
• The scaling factor η for knowledge updating is highly sensitive; defaulting to 1e-6 is critical for optimal performance across all metrics.

Evidence (verbatim from paper)

For the reconstruction task, we aim to reconstruct the target concept directly from the knowledge itself. Consequently, we evaluate concept fidelity using CLIP-I and CLIP-I-Seg. CLIP-I calculates the average pairwise cosine similarity between the CLIP embeddings of the generated and real images. Since background elements in generated images can affect fidelity scores, we also utilize CLIP-I-Seg. This metric employs SAM3 to segment the target concept from the generated images before calculating the similarity with the real images. For the generation task, we aim to integrate updated knowledge with other prompts to enable flexible customized generation. We assess prompt fidelity and human preference in addition to concept fidelity. We measure prompt fidelity using CLIP-T, which computes the average cosine similarity between prompt and image embeddings. We evaluate human preference using Pick Score.

Citation

@misc{zhu2026mokus,
  title={MoKus: Leveraging Cross-Modal Knowledge Transfer for Knowledge-Aware Concept Customization},
  author={Zhu et al. (2026)},
  year={2026},
  note={arXiv:2603.12743}
}

• arXiv: 2603.12743