Black Box LLM Granularity Eval

black-box-llm-granularity-eval

Enabling Fine-Grained Operating Points for Black-Box LLMs — Beyazit et al. (2025) (arXiv:2510.17727, 2025)

What this evaluates

This evaluation probes a black-box LLM's ability to generate high-cardinality, continuous probability scores for binary classification tasks. It measures how well different prompting and post-processing methods improve operational granularity (control over precision-recall operating points) while maintaining predictive performance.

Datasets

• 11 binary classification datasets (combined into a joint dataset for one experiment) — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• PRAUC (primary) — range: [0, 1]
  • Area under the Precision-Recall curve. Computed by calculating precision and recall at every unique predicted score threshold, then integrating the resulting curve. Values range from 0 to 1, with higher being better.
• output_cardinality — range: other
  • Number of unique predicted scores (|ŷ|) generated across the dataset. Measures the diversity of the output distribution.
• g^{pre}, g^{rec}, g^{fpr} — range: other
  • Granularity metrics measuring the spread/concentration of operating points along the precision, recall, and false positive rate axes respectively. Calculated per Equation 2 in the paper. Lower values indicate better granularity.

Input / output format

Input: Text prompts containing binary classification instances (features/context) passed to black-box LLM APIs.

Output: Verbalized probability scores (e.g., "0.75") or class predictions. The evaluation focuses on methods that output continuous scores for thresholding.

Scoring recipe

def compute_prauc(y_true, y_scores):
    desc_indices = np.argsort(y_scores)[::-1]
    y_true_sorted = np.array(y_true)[desc_indices]
    tp = np.cumsum(y_true_sorted)
    fp = np.cumsum(1 - y_true_sorted)
    precision = tp / (tp + fp + 1e-8)
    recall = tp / (tp[-1] + 1e-8)
    return np.trapz(precision, recall)

Common pitfalls

• The proposed methods do not necessarily improve raw classification accuracy (e.g., F1 or AUROC); their main contribution is increasing score granularity for operating point selection.
• Baseline sampling methods (Sample-Class, Sample-Prob) require 20 LLM calls per instance, making them computationally expensive compared to the proposed 1-2 call methods.
• Individual datasets are small (some ≤250 samples) and have nearly monotonic PR curves, which limits the observable performance gains of supervised methods on them alone.

Evidence (verbatim from paper)

We observe that compared to Prompt-Naive, proposed method significantly improves the operational granularity by increasing the diversity and cardinality of the outputs, while outperforming it in terms of PRAUC.

Citation

@misc{beyazit2025enabling,
  title={Enabling Fine-Grained Operating Points for Black-Box LLMs},
  author={Beyazit et al. (2025)},
  year={2025},
  note={arXiv:2510.17727}
}

• arXiv: 2510.17727