Ref Avs Eval

ref-avs-eval

PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing — Song et al. (2026) (arXiv:2604.05018, 2026)

What this evaluates

Evaluates a model's ability to segment objects in audio-visual videos based on natural language referring expressions. It tests both seen categories and generalization to unseen categories, as well as handling null references where no object exists.

Datasets

• Ref-AVS Dataset — total 4000; splits: train (2908), val (276), test (818)

Metrics

• Jaccard Index ($\mathcal{J}$) (primary) — range: [0, 1]
  • Intersection over union of predicted and ground truth binary masks.
• F-score ($\mathcal{F}$) — range: [0, 1]
  • Harmonic mean of precision and recall computed on binary mask predictions.
• S — range: other
  • Square root of the ratio of predicted mask area to background area: $S = \sqrt{\text{predicted mask area/background area}}$. Lower values indicate better performance.

Input / output format

Input: 10-second video frames, corresponding audio, and a natural language text expression referring to a target object.

Output: Pixel-level segmentation mask for the target object.

Scoring recipe

def compute_j(pred, gt):
    inter = (pred & gt).sum()
    union = (pred | gt).sum()
    return inter / union if union > 0 else 0.0

def compute_f(pred, gt):
    tp = (pred & gt).sum()
    fp = (pred & ~gt).sum()
    fn = (~pred & gt).sum()
    p = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    r = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * p * r / (p + r) if (p + r) > 0 else 0.0

def compute_s(pred, gt):
    pred_area = pred.sum()
    bg_area = gt.size - gt.sum()
    return np.sqrt(pred_area / bg_area) if bg_area > 0 else 0.0

Common pitfalls

• The S metric is only evaluated on the Null subset, not the Seen/Unseen subsets.
• Lower S values indicate better performance, which is inverse to standard accuracy metrics.
• The test set is strictly partitioned into Seen, Unseen, and Null subsets, requiring separate evaluation rather than a single aggregate score.

Evidence (verbatim from paper)

We employed the Jaccard Index $(\mathcal{J})$ and F-score $(\mathcal{F})$ for the Seen and Unseen subsets. Null Subset Metric: We employed the metric $S$, which measures the ratio of the predicted mask area to the background area ( $S = \sqrt{\text{predicted mask area/background area}}$ ). A lower $S$ value indicates better performance (less incorrect segmentation).

Citation

@misc{song2026paperorchestra,
  title={PaperOrchestra: A Multi-Agent Framework for Automated AI Research Paper Writing},
  author={Song et al. (2026)},
  year={2026},
  note={arXiv:2604.05018}
}

• arXiv: 2604.05018