FastVideo Model Porting & Alignment

FastVideo Model Porting & Alignment

Goal

{{goal}}

Inputs

• Model name: {{model_name}}
• Official repo: {{official_repo}}
• Official weights path: {{official_weights_path}}

If an input is missing, infer a sensible value and continue.

Source of Truth

Read and follow:

1. docs/contributing/coding_agents.md
2. docs/design/overview.md
3. docs/contributing/testing.md

Reference:

• https://haoailab.com/FastVideo/contributing/coding_agents/#faq

Non-Negotiable Workflow

1. Isolate workspace first (worktree if already in FastVideo; clone otherwise).
2. Set isolated directory as default task root.
3. Fetch official code + weights.
4. Write PLAN.md before implementation.
5. Execute PLAN.md step-by-step.
6. Treat parity/alignment as first-class acceptance criteria.

Never jump from idea to direct edits.

Step 0: Workspace Isolation (Required)

If current directory is a FastVideo repo

MODEL_SLUG="${MODEL_NAME:-{{model_name}}}"
ROOT="$(git rev-parse --show-toplevel)"
STAMP="$(date +%Y%m%d-%H%M%S)"
BRANCH="codex/port-${MODEL_SLUG}-${STAMP}"
WT_DIR="$(dirname "$ROOT")/FastVideo-${MODEL_SLUG}-${STAMP}"

git -C "$ROOT" worktree add -b "$BRANCH" "$WT_DIR"
cd "$WT_DIR"

If current directory is NOT a FastVideo repo

MODEL_SLUG="${MODEL_NAME:-{{model_name}}}"
STAMP="$(date +%Y%m%d-%H%M%S)"
TARGET_DIR="$HOME/dev/FastVideo-${MODEL_SLUG}-${STAMP}"

git clone https://github.com/hao-ai-lab/FastVideo.git "$TARGET_DIR"
cd "$TARGET_DIR"
git checkout -b "codex/port-${MODEL_SLUG}-${STAMP}"

From this point on, use the isolated directory as the root for all commands.

Step 0.5: Fetch Official Repo + Weights

• Clone the official implementation under the FastVideo root.
• Download official weights to official_weights/<model_name>/.
• If a valid Diffusers-format repo exists, prefer direct download.

Example:

python scripts/huggingface/download_hf.py \
  --repo_id "{{official_repo}}" \
  --local_dir "official_weights/{{model_name}}" \
  --repo_type model

Step 1: Plan First (PLAN.md)

Create PLAN.md with checkboxes and acceptance criteria.

Minimum sections:

1. Scope and constraints
2. Architecture mapping (official -> FastVideo)
3. Component parity milestones (DiT first)
4. Pipeline wiring milestones
5. Testing milestones (component parity, pipeline parity, SSIM)
6. Example + docs milestones
7. Risks and mitigations

Step 2: Implement in Parity-First Order

2.1 Model + mapping first

• Add/extend model in fastvideo/models/...
• Add config + param_names_mapping in fastvideo/configs/models/...
• Reuse existing FastVideo layers where possible
• Attention convention:
  • DistributedAttention for full-sequence self-attention in DiT
  • LocalAttention for cross-attention / non-global attention

FAQ rule: implement FastVideo model shape/naming first, then finalize conversion/mapping.

2.2 Numerical alignment immediately

• Add component parity tests under tests/local_tests/...
• Compare official vs FastVideo outputs with fixed seeds/inputs
• Start with atol=1e-4, rtol=1e-4
• Keep dtype consistent (bf16 if available, else fp32)
• Use load_state_dict(strict=False) while iterating mapping

If parity fails, debug in this order:

1. Fix key mapping (param_names_mapping)
2. Align attention backend (for example FASTVIDEO_ATTENTION_BACKEND=TORCH_SDPA)
3. Align scheduler/sigma/timestep behavior
4. Add activation logging to locate first divergence

2.3 Repeat for each component

Repeat model+mapping+parity for each required component:

• DiT
• VAE
• Encoder/tokenizer
• Any model-specific extras

2.4 Pipeline integration

• Add pipeline config in fastvideo/configs/pipelines/
• Add sampling defaults in fastvideo/configs/sample/
• Register via explicit register_configs(...) in fastvideo/registry.py
• Add pipeline logic in fastvideo/pipelines/basic/<pipeline>/
• Add reusable stages in fastvideo/pipelines/stages/

2.5 End-to-end validation

• Add pipeline parity tests: tests/local_tests/pipelines/
• Add SSIM tests: fastvideo/tests/ssim/
• Add minimal runnable example: examples/inference/basic/
• Run locally and generate a video sample

2.6 Documentation

Update docs/ with:

• usage
• constraints
• memory/speed caveats
• backend requirements

Diffusers vs Conversion Rule

If Diffusers-format exists and loads correctly:

• skip conversion script
• focus on mapping + parity

If not:

• add conversion script
• stage converted output at converted_weights/<model_name>/
• still validate parity against official implementation

Alignment Gate (Must Pass)

• Mapping rules explicit and reviewed
• Missing/unexpected key mismatches resolved
• Component parity tests passing
• Pipeline parity checks passing
• SSIM regression checks passing
• Example script generates expected video
• Documentation updated

Required Output

When using this skill, always provide:

1. Isolated workspace path
2. Branch name
3. Completed PLAN.md
4. Execution progress per plan item
5. Final parity/test summary + residual risks