Maggie Instance Matting Eval

maggie-instance-matting-eval

MaGGIe: Masked Guided Gradual Human Instance Matting — Huynh et al. (2024) (arXiv:2404.16035, 2024)

What this evaluates

Evaluates the capability of models to generate high-fidelity alpha mattes for multiple human instances in images and videos, focusing on detail preservation, instance separation, and temporal consistency across frames.

Datasets

• HIM2K+M-HIM2K — total ?; splits: test (-1)
• V-HIM60 — total ?; splits: test (-1)

Metrics

• MAD (primary) — range: [0, 1]
  • Mean Absolute Difference between predicted alpha matte and ground truth alpha over valid pixels. Lower is better.
• MSE — range: [0, 1]
  • Mean Squared Error between predicted and ground truth alpha values. Lower is better.
• Grad — range: [0, 1]
  • Mean absolute difference between the gradient of the predicted alpha and the ground truth alpha, measuring detail preservation.
• Conn — range: [0, 1]
  • Connectivity error measuring the structural consistency and foreground region integrity of the alpha matte.
• dtSSD — range: other
  • Difference in temporal Sum of Squared Differences between consecutive frames' alpha mattes, assessing temporal consistency.
• MESSDdt — range: other
  • Mean Error in Sum of Squared Differences temporal metric, evaluating frame-to-frame stability.

Input / output format

Input: RGB image or video frame paired with instance mask guidance (binary or soft masks per instance).

Output: Per-instance alpha matte prediction (single-channel float tensor per instance).

Scoring recipe

def evaluate(preds, gts):
    instance_metrics = []
    for pred, gt in zip(preds, gts):
        m = {}
        m['MAD'] = np.mean(np.abs(pred - gt))
        m['MSE'] = np.mean((pred - gt) ** 2)
        m['Grad'] = np.mean(np.abs(grad(pred) - grad(gt)))
        m['Conn'] = connectivity_error(pred, gt)
        instance_metrics.append(m)
    avg = {k: np.mean([inst[k] for inst in instance_metrics]) for k in instance_metrics[0]}
    return avg

Common pitfalls

• Metrics are averaged per instance before final aggregation, not computed on averaged predictions.
• Comparisons with public baselines (e.g., InstMatt) may be unfair if they were trained on private external data not retrained on the benchmark.
• Instance detection metrics like IMQ are explicitly excluded; evaluation focuses solely on alpha matte quality per known instance.

Evidence (verbatim from paper)

Our evaluation metrics included Mean Absolute Differences (MAD), Mean Squared Error (MSE), Gradient (Grad), and Connectivity (Conn). We also separately computed these metrics for the foreground and unknown regions, denoted as MADf and MADu, by estimating the trimap on the ground truth. Since our images contain multiple instances, metrics were calculated for each instance individually and then averaged.

Citation

@misc{huynh2024maggie,
  title={MaGGIe: Masked Guided Gradual Human Instance Matting},
  author={Huynh et al. (2024)},
  year={2024},
  note={arXiv:2404.16035}
}

• arXiv: 2404.16035