# Cab Eval

> Benchmarks LLM bias by scoring responses to automatically generated open-ended questions across sensitive attributes, producing a composite fitness score from 0 to 5.

- Skill: `qhjqhj00/cab-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/cab-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/cab-eval/raw
- Safety review: PASS (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML, Research & Search, Model Training & Fine-tuning, Summarization
- Tags: Benchmark, Bias Evaluation, Cab, Fitness Score, Llm Bias, Open Ended Questions
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-08-22
- Page: https://skillmd.com/skills/qhjqhj00/cab-eval

---


# cab-eval

> Adaptive Generation of Bias-Eliciting Questions for LLMs — Staab et al. (2025) (arXiv:2510.12857, 2025)

## What this evaluates

Evaluates whether LLMs exhibit biased responses to automatically generated, realistic open-ended questions. It probes for hidden biases across sensitive attributes (e.g., sex, race, religion) by measuring asymmetric refusals, explicit acknowledgments, and other bias dimensions.

## Datasets

- **CAB** — total ?; splits: explicit (-1), implicit (-1); repo https://github.com/eth-sri/cab

## Metrics

- `fitness score` **(primary)** — range: [0, 5]
  - A composite score ranging from 0 to 5, computed by combining four dimensions of bias (including asymmetric refusals and explicit acknowledgment) as described in Section 3. Higher scores indicate stronger bias exhibited by the model.

## Input / output format

**Input**: Automatically generated, realistic open-ended questions/prompts designed to elicit biased responses, categorized by sensitive attributes (e.g., sex, race, religion).

**Output**: The model's natural language response to the generated question.

## Scoring recipe

```python
def compute_fitness_score(predictions, gold):
    scores = []
    for prompt, response in zip(predictions, gold):
        dim_scores = judge_model.score_bias_dimensions(response)
        fitness = combine_dimensions(dim_scores)  # range [0, 5]
        scores.append(fitness)
    return mean(scores)
```

## Common pitfalls

- The metric is a composite score from an LLM judge, not a standard NLP metric like accuracy or BLEU.
- The benchmark distinguishes between explicit and implicit question subsets, which yield significantly different bias elicitation rates (~40% drop for implicit).
- Scores are bounded [0, 5], but most models average below 1, indicating generally low bias despite high elicitation rates (>65% of sex questions triggered bias in at least one model).

## Evidence (verbatim from paper)

> The fitness scores are computed as described in [Section˜3], combining the four dimensions of bias into a single value. Higher fitness scores indicate stronger bias exhibited by a model in response to CAB’s questions. ... we note that the range of fitness scores is [0-5] with no model having an average above 1

## Citation

```bibtex
@misc{staab2025adaptive,
  title={Adaptive Generation of Bias-Eliciting Questions for LLMs},
  author={Staab et al. (2025)},
  year={2025},
  note={arXiv:2510.12857}
}
```

- arXiv: 2510.12857