Scendi Score

scendi-score

Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddings — Ospanov et al. (2024) (arXiv:2412.18645, 2024)

What this evaluates

Evaluates the intrinsic diversity of text-to-image generative models by isolating model-driven variation from prompt-driven variation. It uses CLIP embeddings to construct a joint image-text kernel covariance matrix and applies Schur complement decomposition to remove text influence before computing spectral entropy.

Datasets

• Cat breeds dataset — total ?; splits: test (-1)

Metrics

• Scendi score (primary) — range: other (≥ 1)
  • Computes the Schur complement of the image-text kernel covariance matrix to isolate model-driven variation: $\Lambda_I = C_{II} - C_{IT}C_{TT}^{-1}C_{IT}^\top$. Takes the eigenvalues $\lambda_j$ of $\Lambda_I$, computes the trace $\text{Tr}(\Lambda_I)$, and returns $\exp(\sum_j \lambda_j \log(\text{Tr}(\Lambda_I)/\lambda_j))$, which corresponds to the exponential of the conditional Shannon entropy of image clusters given the prompt.

Input / output format

Input: A paired dataset of $n$ text prompts and corresponding generated images $(T_j, I_j)$.

Output: A single scalar diversity score.

Scoring recipe

# 1. Get CLIP embeddings for images (Phi_I) and texts (Phi_T)
# 2. Compute covariance matrices
C_II = (Phi_I.T @ Phi_I) / n
C_IT = (Phi_I.T @ Phi_T) / n
C_TT = (Phi_T.T @ Phi_T) / n
# 3. Compute Schur complement (model-driven component)
Lambda_I = C_II - C_IT @ np.linalg.inv(C_TT) @ C_IT.T
# 4. Compute eigenvalues and trace
eigenvalues = np.linalg.eigvalsh(Lambda_I)
trace_Lambda = np.sum(eigenvalues)
# 5. Compute Scendi score
score = np.exp(np.sum(eigenvalues * np.log(trace_Lambda / eigenvalues)))
return score

Common pitfalls

• Forgetting to subtract the text-driven component ($C_{IT}C_{TT}^{-1}C_{IT}^\top$) before computing eigenvalues, which conflates prompt diversity with model diversity.
• Using raw covariance eigenvalues without accounting for the trace normalization in the entropy formula, leading to scale-dependent scores.
• Applying the metric to unpaired or mismatched image-text datasets, as the Schur complement decomposition strictly requires aligned $(T_j, I_j)$ pairs.

Evidence (verbatim from paper)

We define the Schur-Complement-ENtropy Diversity ($\mathrm{Scendi}$) score as follows: $\displaystyle\mathrm{Scendi}(x_{1},..,x_{n};t_{1},..,t_{n}),:=$ $\displaystyle,\exp\Bigl{(}\sum_{j=1}^{d}\lambda^{(\Lambda_{I})}{j}\log\frac{\mathrm{Tr}(\Lambda{I})}{\lambda^{(\Lambda_{I})}{j}}\Bigr{)}$ where $\lambda^{(\Lambda{I})}{j}$ denotes the $j$th eigenvalue of matrix $\Lambda{I}$ and $\mathrm{Tr}(\Lambda_{I})=\sum_{j=1}^{d}\lambda^{(\Lambda_{I})}_{j}$ is the sum of the eigenvalues.

Citation

@misc{ospanov2024scendi,
  title={Scendi Score: Prompt-Aware Diversity Evaluation via Schur Complement of CLIP Embeddings},
  author={Ospanov et al. (2024)},
  year={2024},
  note={arXiv:2412.18645}
}

• arXiv: 2412.18645