Step Audio Editx Eval

step-audio-editx-eval

Step-Audio-EditX Technical Report — Yan et al. (2025) (arXiv:2511.03601, 2025)

What this evaluates

Probes a model's capability to accurately edit or synthesize audio with specific emotional tones, speaking styles, and paralinguistic elements (e.g., laughter, sighs) using zero-shot voice cloning or iterative refinement. It measures how well the model preserves linguistic content while transferring or inserting target audio attributes.

Datasets

• Step-Audio-Edit-Test — total ?; splits: test (-1); repo https://github.com/stepfun-ai/Step-Audio-EditX

Metrics

• accuracy (primary) — range: percent
  • Percentage of correctly classified audio samples out of the total, averaged across all speakers and target categories (5 emotions, 7 styles).
• paralinguistic_score — range: [1, 3]
  • Average score assigned by the Gemini-2.5-Pro LLM judge on a 1–3 scale (3 = perfect, 2 = flawed, 1 = failed) for paralinguistic insertion accuracy.

Input / output format

Input: Generated audio clips (from zero-shot voice cloning or closed-source TTS) paired with the target instruction or category label (e.g., emotion, style, or paralinguistic tag).

Output: Categorical classification label for emotion/style tasks; integer score from 1 to 3 for paralinguistic tasks.

Scoring recipe

# Emotion/Style Accuracy
correct = sum(1 for pred, gold in zip(predictions, gold_labels) if pred == gold)
accuracy = (correct / len(gold_labels)) * 100

# Paralinguistic Score
llm_scores = [gemini_judge_score(audio, instruction) for audio, instruction in zip(audios, instructions)]
paralinguistic_score = sum(llm_scores) / len(llm_scores)

Common pitfalls

• LLM-as-a-judge (Gemini-2.5-Pro) may favor fluent but content-less audio, leading to inflated scores for paralinguistic omission.
• Iterative editing setup (N=3) means final scores depend heavily on the prompt audio provided in each step; holding prompt audio constant significantly alters results.
• Baseline 'Iter0' scores reflect zero-shot cloning capability, not editing performance, and should not be conflated with the model's editing accuracy.

Evidence (verbatim from paper)

We introduce Step-Audio-Edit-Test, a benchmark that leverages LLM-as-a-judge model to evaluate model performance on emotion, speaking style, and paralinguistics. All evaluation audio is generated via zero-shot voice cloning and subsequently scored using the Gemini-2.5-Pro model... The final accuracy for each category is calculated as the average across all speakers... employing a rigorous 1–3 scoring scale (3 = perfect, 2 = flawed, 1 = failed)... Finally, model performance in the paralinguistic editing task is assessed by calculating the overall average score generated by Gemini-2.5-Pro model.

Citation

@misc{yan2025stepaudioeditx,
  title={Step-Audio-EditX Technical Report},
  author={Yan et al. (2025)},
  year={2025},
  note={arXiv:2511.03601}
}

• arXiv: 2511.03601