Featureid 3ds Eval

featureid-3ds-eval

Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control — Armitage et al. (2025) (arXiv:2512.24826, 2025)

What this evaluates

Evaluates a derivative-free camera control policy's ability to align vision-language models in 3D multi-object scenes. It probes robustness to viewpoint changes and object occlusions using minimal demonstration data.

Datasets

• FeatureID-3DS — total ?; splits: test (-1)
• PartialView-3DS — total ?; splits: test (-1)

Metrics

• prediction error (primary) — range: other
  • Computed over viewpoints where errors were marked in demonstrations and during the measurement round.

Input / output format

Input: 3D scene coordinates (X, Y viewpoints, z-axis levels), demonstration data with marked errors, and noisy VLM outputs.

Output: Predicted camera actions for the correction round and updated demonstration data with system decisions.

Scoring recipe

# Pseudo-code based on section description
def evaluate(demonstrations, vlm_outputs, camera_state):
    # Measurement round
    system_decisions = update_demonstration_data(demonstrations, camera_state)
    coefficients = measure_coefficients(system_decisions, viewpoint_labels)
    
    # Correction round
    predicted_actions = predict_camera_actions(coefficients)
    
    # Scoring
    prediction_errors = compute_errors(predicted_actions, marked_viewpoints)
    return prediction_errors

Common pitfalls

• Relies on external VLMs (llava-v1.5-13b, vicuna-13b-v1.5) without specifying exact prompt templates or inference parameters.
• Uses only 5% of scenes (n=3) for demonstrations, which may not capture full scene diversity.
• Hardware setup splits VLM inference (A100) and camera control (RTX 2080), but latency/throughput metrics are not reported in this section.

Evidence (verbatim from paper)

Methods receive prediction errors for viewpoints where an error was marked in demonstrations and during the measurement round. Demonstrations are set at 5% of scenes for each benchmark: n = 3 for feature identification (FeatureID-3DS) and object occlusion (PartialView-3DS).

Citation

@misc{armitage2025video,
  title={Video and Language Alignment in 2D Systems for 3D Multi-object Scenes with Multi-Information Derivative-Free Control},
  author={Armitage et al. (2025)},
  year={2025},
  note={arXiv:2512.24826}
}

• arXiv: 2512.24826