Gamefactory Eval

gamefactory-eval

GameFactory: Creating New Games with Generative Interactive Videos — Jiwen Yu et al. (2025) (arXiv:2501.08325, 2025)

What this evaluates

This evaluation protocol assesses a video generation model's ability to follow discrete and continuous action inputs while maintaining semantic alignment with text prompts and preserving the original model's visual domain. It measures action-following accuracy, camera pose consistency, text-video semantic relevance, and overall video generation quality across in-domain and open-domain scenes.

Datasets

• GF-Minecraft — total ?; splits: test (-1)
• VPT (Find Cave) — total ?; splits: test (-1)

Metrics

• Flow (primary) — range: other
  • Mean squared error between the optical flow of the generated video and the reference video. Lower values indicate better action-following dynamics.
• Cam — range: other
  • Euclidean distance between camera poses extracted from predicted and reference videos using GLOMAP. Lower values indicate better camera pose consistency.
• CLIP — range: [0, 1]
  • Cosine similarity between CLIP features of the generated video and the input text prompt. Higher values indicate better semantic relevance.
• FID — range: other
  • Fréchet Inception Distance measuring the distribution difference between generated and reference videos. Lower values indicate better generation quality.
• FVD — range: other
  • Fréchet Video Distance measuring the distribution difference between generated and reference videos. Lower values indicate better temporal generation quality.

Input / output format

Input: Text prompt describing the scene/game, combined with an action control signal (discrete keyboard keys or continuous mouse coordinates).

Output: A sequence of video frames at 360x640 resolution.

Scoring recipe

def compute_metrics(pred_video, ref_video, text_prompt):
    flow_err = mse(optical_flow(pred_video), optical_flow(ref_video))
    cam_dist = euclidean_dist(glomap_pose(pred_video), glomap_pose(ref_video))
    clip_sim = cosine_similarity(clip_encode(pred_video), clip_encode(text_prompt))
    fid = frechet_inception_distance([pred_video], [ref_video])
    fvd = frechet_video_distance([pred_video], [ref_video])
    return {'Flow': flow_err, 'Cam': cam_dist, 'CLIP': clip_sim, 'FID': fid, 'FVD': fvd}

Common pitfalls

• Metrics have different optimization directions: Flow, Cam, FID, and FVD are lower-is-better, while CLIP is higher-is-better.
• Evaluating on human-biased datasets (e.g., VPT) instead of uniformly distributed action datasets leads to poor generalization on rare or counter-intuitive actions like moving backward or jumping in place.
• Confusing in-domain evaluation (training distribution) with open-domain scene generalization, which requires decoupled training to avoid style leakage.

Evidence (verbatim from paper)

We use these evaluation metrics: (1) Flow: calculates the optical flow of the generated video to reflect its dynamics, assessing action-following ability by measuring mean square error to the optical flow of the reference video; (2) Cam: computes the Euclidean distance between camera poses extracted from predicted videos and those extracted from reference videos, where both sets of camera poses are obtained using GLOMAP*[[32]]. (3) CLIP: computes feature similarity in the CLIP*[[37]]* space to evaluate semantic relevance to the given text prompt; (4) FID, FVD: measures distribution differences between the generated videos and the reference, providing an assessment of generation quality.

Citation

@misc{yu2025gamefactory,
  title={GameFactory: Creating New Games with Generative Interactive Videos},
  author={Jiwen Yu et al. (2025)},
  year={2025},
  note={arXiv:2501.08325}
}

• arXiv: 2501.08325