Im Promptu Eval

im-promptu-eval

Im-Promptu: In-Context Composition from Image Prompts — Bhishma Dedhia et al. (arXiv:2305.17262, 2023)

What this evaluates

Evaluates an agent's ability to perform in-context compositional reasoning from image prompts by generalizing learned primitive relations to unseen source-target pairs and complex composite tasks.

Datasets

• 3D Shapes — total ?; splits: train (-1), test (-1)
• BitMoji Faces — total ?; splits: train (-1), test (-1)
• CLEVR Objects — total ?; splits: train (-1), test (-1)

Metrics

• MSE (primary) — range: other
  • Mean squared error between the pixel values of the generated image and the ground truth image. Lower values indicate better reconstruction fidelity.
• FID — range: other
  • Fréchet Inception Distance measuring perceptual quality by computing the Fréchet distance between the multivariate Gaussian distributions of Inception features for generated and real images. Lower values indicate better perceptual quality.

Input / output format

Input: Image prompts consisting of object components or context images representing source-target pairs.

Output: A generated composite image matching the target composition.

Scoring recipe

def compute_mse(gen_img, gt_img):
    return np.mean((gen_img - gt_img) ** 2)

def compute_fid(gen_imgs, real_imgs):
    gen_feats = inception_model(gen_imgs)
    real_feats = inception_model(real_imgs)
    mu_gen, sigma_gen = np.mean(gen_feats, axis=0), np.cov(gen_feats, rowvar=False)
    mu_real, sigma_real = np.mean(real_feats, axis=0), np.cov(real_feats, rowvar=False)
    diff = mu_gen - mu_real
    covmean, _ = scipy.linalg.sqrtm(sigma_gen.dot(sigma_real), disp=False)
    return np.real(np.sum(diff**2) + np.trace(sigma_gen + sigma_real - 2 * covmean))

Common pitfalls

• MSE can be misleadingly low if models generate blurry or 'blob-like' outputs that match pixel averages but lack visual fidelity.
• FID measures perceptual quality but does not capture structural correctness or object-level composition accuracy.
• Patch-based methods may appear effective on simple datasets but fail to capture inter-object dependencies like occlusions and shadows.

Evidence (verbatim from paper)

Fig. 4(a) plots scores of different agents across benchmarks against two key metrics: (1) MSE (lower is better) that quantitatively compares the construction against the ground truth and (2) Fréchet inception distance (FID, lower is better) score to measure the perceptual quality of the composition.

Citation

@misc{dedhia2023impromptu,
  title={Im-Promptu: In-Context Composition from Image Prompts},
  author={Bhishma Dedhia et al.},
  year={2023},
  note={arXiv:2305.17262}
}

• arXiv: 2305.17262