Text Animator Eval

text-animator-eval

Text-Animator: Controllable Visual Text Video Generation — Liu et al. (2024) (arXiv:2406.17777, 2024)

What this evaluates

Evaluates a text-to-video model's ability to accurately render and animate specific text within a video scene while maintaining visual quality and temporal consistency. It probes character-level text fidelity, resistance to text collapse during motion, and overall video generation quality.

Datasets

• LAION subset — total 90; splits: test (90)

Metrics

• Sen. Acc (primary) — range: [0, 1]
  • Sentence Accuracy. Calculated by cropping each generated text line according to the specified position, feeding it into an OCR model, and checking if the predicted text exactly matches the ground truth. Accuracy = (Number of correctly recognized text lines) / (Total text lines).
• NED — range: [0, 1]
  • Normalized Edit Distance. Measures string similarity between the OCR-predicted text and the ground truth. Computed as 1 - (Levenshtein distance / max(len(pred), len(gold))).
• FID — range: other
  • Fréchet Inception Distance. Computes the distance between the distribution of features extracted from generated video frames and real-world video frames to assess appearance quality.
• Prompt similarity — range: other
  • Evaluates the semantic similarity between the input text prompt and the generated video output, typically using a vision-language embedding model.
• Frame similarity — range: other
  • Measures the temporal continuity and consistency between consecutive frames in the generated video sequence.

Input / output format

Input: Text prompt (caption) concatenated with hint prompts (e.g., 'these texts are written on it: xxx'), along with camera pose and text position control signals.

Output: Video sequence of 16 frames at 256x384 resolution.

Scoring recipe

def compute_metrics(predictions, golds, bboxes):
    correct = 0
    ned_scores = []
    for pred_video, gold_text, bbox in zip(predictions, golds, bboxes):
        pred_text = ocr_model(crop(pred_video, bbox))
        if pred_text == gold_text:
            correct += 1
        dist = levenshtein(pred_text, gold_text)
        max_len = max(len(pred_text), len(gold_text))
        ned_scores.append(1 - (dist / max_len) if max_len > 0 else 1.0)
    sen_acc = correct / len(predictions)
    avg_ned = sum(ned_scores) / len(ned_scores)
    return sen_acc, avg_ned

Common pitfalls

• OCR models often fail on stylized, curved, or heavily distorted text, leading to artificially low Sentence Accuracy scores.
• FID is typically computed on images; applying it to video frames without accounting for temporal redundancy can inflate scores or misrepresent video quality.
• Lack of a standardized benchmark dataset forces reliance on manually curated subsets (~90 images), limiting statistical significance and cross-paper comparability.

Evidence (verbatim from paper)

According to the paper[[27]], we employed the Sentence Accuracy (Sen. Acc) metric, where each generated text line is cropped according to the specified position and fed into an OCR model to obtain predicted results. Additionally, the Normalized Edit Distance (NED)[[18]] is used to measure the similarity between two strings. To ensure that our method has better video generation capabilities, we utilize the Fréchet Inception Distance (FID) to assess the video appearance quality between generated videos and real-world videos. Moreover, we also adopted the Prompt similarity and the Frame similarity metric.

Citation

@misc{liu2024textanimator,
  title={Text-Animator: Controllable Visual Text Video Generation},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2406.17777}
}

• arXiv: 2406.17777