Audio To Image Eval

audio-to-image-eval

Seeing Sound: Assembling Sounds from Visuals for Audio-to-Image Generation — Petermann et al. (2025) (arXiv:2501.05413, 2025)

What this evaluates

Evaluates an audio-to-image generative model's ability to synthesize semantically aligned images from audio prompts. It measures cross-modal alignment, perceptual image quality, and distributional similarity against ground-truth visuals.

Datasets

• Greatest Hits — total ?; splits: test (-1)
• Landscapes — total ?; splits: test (-1)
• Into The Wild (ITW) — total ?; splits: test (-1)
• VEGAS — total ?; splits: test (-1)
• VGGSound — total 1000; splits: test (1000)

Metrics

• Audio-Image Similarity (AIS) — range: [0, 1]
  • Cosine similarity between latent embeddings of the audio prompt and generated image, computed using Wav2CLIP.
• Image-Image Similarity (IIS) — range: [0, 1]
  • Cosine similarity between latent embeddings of the ground-truth and generated images, computed using the CLIP visual tower.
• Fréchet Inception Distance (FID) (primary) — range: [0, ∞)
  • Fréchet distance between the multivariate Gaussian distributions of features extracted from generated and ground-truth images. Lower values indicate better perceptual quality and diversity.

Input / output format

Input: 5-second audio excerpt (processed via AST embeddings) conditioning a diffusion model.

Output: 512×512 RGB image.

Scoring recipe

def evaluate(predictions, gold):
    gen_images = predictions['images']
    gt_images = gold['images']
    audio_clips = gold['audio']
    
    # AIS: cosine similarity via Wav2CLIP
    audio_emb = wav2clip.encode(audio_clips)
    gen_emb = wav2clip.encode(gen_images)
    ais = cosine_similarity(audio_emb, gen_emb).mean()
    
    # IIS: cosine similarity via CLIP visual tower
    gt_emb = clip_vision.encode(gt_images)
    iis = cosine_similarity(gt_emb, gen_emb).mean()
    
    # FID: distribution distance (computed over full dataset)
    fid = frechet_inception_distance(gen_images, gt_images)
    
    return {'AIS': ais, 'IIS': iis, 'FID': fid}

Common pitfalls

• In-sample vs out-of-sample confusion: Models evaluated on datasets they were trained on show inflated metrics due to memorization; the paper highlights out-of-sample results as the fair comparison.
• Metric directionality: FID is lower-is-better, whereas AIS and IIS are higher-is-better. Misinterpreting the sign leads to incorrect model ranking.
• Audio standardization: All audio prompts are standardized to exactly 5 seconds and processed through AST embeddings before conditioning; raw audio lengths vary in source datasets.

Evidence (verbatim from paper)

Following recent literature, we evaluate different audio-to-image generative models using the following metrics. Audio-Image Similarity (AIS) aims at evaluating the alignment between a generated image and its audio counterpart, namely by computing cosine similarity between latent embeddings associated with the two modality instances. In accordance with [1, 43], we also employ Wav2CLIP [40] for this purpose. Image-Image Similarity (IIS) is the image analog to what AIS is for audio and measures the semantic similarity between generated and ground-truth images. We utilize the visual tower of CLIP [15] to obtain latent representations from images when implementing IIS. Finally, we adopt Fréchet Inception Distance (FID) [16] to quantify the distance between distribution of the generated and ground-truth images.

Citation

@misc{petermann2025seeingsound,
  title={Seeing Sound: Assembling Sounds from Visuals for Audio-to-Image Generation},
  author={Petermann et al. (2025)},
  year={2025},
  note={arXiv:2501.05413}
}

• arXiv: 2501.05413