Ft Ncfm Eval

ft-ncfm-eval

FT-NCFM: An Influence-Aware Data Distillation Framework for Efficient VLA Models — Chen et al. (2025) (arXiv:2511.16233, 2025)

What this evaluates

Evaluates the performance and data efficiency of a Vision-Language-Action (VLA) model trained on a synthetically distilled coreset compared to models trained on full datasets. It probes long-horizon manipulation, multi-task skill acquisition, and generalization across spatial, object, goal, and temporal dimensions.

Datasets

• CALVIN — total ?; splits: test (-1)
• Meta-World — total ?; splits: test (-1)
• LIBERO — total ?; splits: test (-1)

Metrics

• Success Rate (SR %) (primary) — range: percent
  • Percentage of successful task executions out of total evaluation trials.
• Average Task Completion Length (Avg. Len) (primary) — range: other
  • Average number of steps or sub-tasks completed successfully in long-horizon evaluations.
• Total Training Time (GPU-hours) — range: other
  • Total GPU-hours required for the model to converge from random initialization, including any one-time preprocessing overhead.

Input / output format

Input: RGB visual observations processed by a ViT-B/16 backbone and natural language instructions.

Output: Continuous or discrete action vectors for robot manipulation.

Scoring recipe

def compute_metrics(trials, total_gpu_seconds):
    success_count = sum(1 for t in trials if t.success)
    sr = (success_count / len(trials)) * 100
    
    total_len = sum(t.completed_steps for t in trials)
    avg_len = total_len / len(trials)
    
    time_hours = total_gpu_seconds / 3600
    
    return {'SR %': sr, 'Avg. Len': avg_len, 'Time (GPU-h)': time_hours}

Common pitfalls

• Baseline training times for policy distillation methods (e.g., RLDG, DROC) include both teacher model training and distillation, which must be accounted for when comparing against the data-centric FT-NCFM overhead.
• The FT engine's preprocessing cost is a one-time investment that should be amortized over multiple training iterations, not treated as a per-epoch cost.
• Performance scaling is evaluated at specific data ratios (1%, 5%, 10%); results are not directly comparable across different compression rates without normalization.

Evidence (verbatim from paper)

We focus on two core metrics: Success Rate (SR %) or Average Task Completion Length (Avg. Len) to measure performance, and Total Training Time (GPU-hours) to measure efficiency.

Citation

@misc{chen2025ftncfm,
  title={FT-NCFM: An Influence-Aware Data Distillation Framework for Efficient VLA Models},
  author={Chen et al. (2025)},
  year={2025},
  note={arXiv:2511.16233}
}

• arXiv: 2511.16233