Emid Emotional Alignment Eval

emid-emotional-alignment-eval

EMID: An Emotional Aligned Dataset in Audio-Visual Modality — Zou et al. (2023) (arXiv:2308.07622, 2023)

What this evaluates

Human-subject validation of cross-modal emotional alignment between music and images. It probes whether pairing audio and visual stimuli based on a 13-dimensional emotional coordinate space yields higher perceptual matching accuracy compared to semantic-only or random matching.

Datasets

• EMID — total ?; splits: test (91); repo https://github.com/ecnu-aigc/EMID

Metrics

• accuracy (primary) — range: [0, 1]
  • Ratio of correct matches (subjects' answers matching the EMID's designated best pair) to the total number of validation questions. Calculated separately for music-to-image, image-to-music, and per emotional category.

Input / output format

Input: Per trial: (1) Music-to-Image: One music clip. (2) Image-to-Music: Three images. Subjects select the best matching image or music clip from four options.

Output: Subject's selected option index (1–4) per trial.

Scoring recipe

def compute_accuracy(predictions, gold):
    correct = sum(1 for p, g in zip(predictions, gold) if p == g)
    return correct / len(gold)

Common pitfalls

• The evaluation uses a small subset (91 trials) rather than the full dataset, so results may not generalize to the entire 30k+ corpus.
• Category-level accuracy is skewed by dataset imbalance; for example, the 'disgusting' category was removed from the source image dataset, affecting category C results.
• The 'correct' answer is pre-determined by the EMID's score matrix rather than majority human vote, which may bias results if the dataset's alignment diverges from general human perception.

Evidence (verbatim from paper)

Based on the response data of different subjects, we calculate the separate accuracy and overall accuracy of music-to-image and image-to-music experiments. Additionally, we conduct an analysis to investigate the distribution of accuracy across different music emotional categories. Note that accuracy refers to the ratio of the number of consistent situations between the subjects' answers and pairs in the EMID to the total number of validation questions.

Citation

@misc{zou2023emid,
  title={EMID: An Emotional Aligned Dataset in Audio-Visual Modality},
  author={Zou et al. (2023)},
  year={2023},
  note={arXiv:2308.07622}
}

• arXiv: 2308.07622