Counterfactual Image Gen Eval

counterfactual-image-gen-eval

Benchmarking Counterfactual Image Generation — Melistas et al. (2024) (arXiv:2403.20287, 2024)

What this evaluates

Evaluates the capability of generative models to produce counterfactual images that preserve causal consistency, maintain realism, and minimally alter non-target attributes under specified interventions. It probes composition stability, attribute manipulation effectiveness, and distributional fidelity across varying dataset complexities and causal graphs.

Datasets

• MorphoMNIST — total ?; splits: test (-1)
• CelebA — total ?; splits: test (-1)
• ADNI — total ?; splits: test (-1)

Metrics

• l1 distance — range: other
  • Mean absolute pixel-wise difference between the original factual image and its reconstructed/counterfactual version. Lower is better.
• LPIPS — range: [0, 1]
  • Learned Perceptual Image Patch Similarity using a VGG-16 backbone. Measures perceptual distance between images. Lower is better.
• MAE — range: other
  • Mean Absolute Error between the predicted value of a continuous causal variable (from a trained regressor) and its ground truth value after intervention. Lower is better.
• Accuracy — range: [0, 1]
  • Proportion of correctly classified categorical variables (e.g., digits) by a trained classifier on the generated counterfactual images. Higher is better.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for binary attribute classifiers (e.g., smiling, eyeglasses) applied to counterfactual images. Higher is better.
• FID (primary) — range: other
  • Fréchet Inception Distance measuring the distance between feature distributions of generated counterfactual images and real images. Lower is better.
• CLD (primary) — range: other
  • Counterfactual Latent Distance measuring minimality of changes in the latent space. Lower is better.

Input / output format

Input: Factual image, causal graph structure, and intervention specification (e.g., do(variable=value)).

Output: Generated counterfactual image.

Scoring recipe

def evaluate(predictions, gold, metric):
    if metric == 'l1':
        return np.mean(np.abs(predictions - gold))
    elif metric == 'LPIPS':
        return lpips_score(predictions, gold, net_type='vgg')
    elif metric == 'MAE':
        pred_val = regressor.predict(predictions)
        return np.mean(np.abs(pred_val - gold))
    elif metric in ['Accuracy', 'F1 score']:
        pred_label = classifier.predict(predictions)
        if metric == 'Accuracy':
            return np.mean(pred_label == gold)
        else:
            return f1_score(gold, pred_label)
    elif metric == 'FID':
        return compute_fid(predictions, gold)
    elif metric == 'CLD':
        return compute_cld(predictions, gold)

Common pitfalls

• LPIPS is explicitly preferred over l1 distance for high-content datasets (CelebA, ADNI) as l1 is not informative for natural images.
• HVAE models may ignore conditioning without the specific counterfactual training process; naive conditioning leads to poor effectiveness.
• Interventions must follow the mutilated causal graph (do-calculus); failing to sever incoming edges to intervened variables breaks causal fidelity.
• CLD and FID trade-offs vary by dataset complexity; GANs may score better on FID for complex graphs but fail on minimality (CLD).

Evidence (verbatim from paper)

To quantitatively evaluate composition, we perform a null-intervention: we abduct the posterior noise and use it to perform inference without intervening on any variable. Following the protocol of (35), we apply composition for 1 and 10 cycles, measuring the distance between the initial observable image and its first and tenth reconstructed versions, respectively. In addition to the l1 distance in the pixel space, we also use the LPIPS (63) metric on VGG-16, since we find that the former is not informative for higher content datasets (e.g., natural images here). The FID metric in Table 10 aligns with our qualitative results: HVAE outperformed both VAE and GAN on MorphoMNIST, CelebA (simple graph) and ADNI. In terms of minimality (CLD metric) (Table 10), HVAE was the best performing model on CelebA (simple graph) and ADNI...

Citation

@misc{melistas2024benchmarking,
  title={Benchmarking Counterfactual Image Generation},
  author={Melistas et al. (2024)},
  year={2024},
  note={arXiv:2403.20287}
}

• arXiv: 2403.20287