# Ept Benchmark Eval

> ept-benchmark-eval

- Skill: `qhjqhj00/ept-benchmark-eval` (Agent Skill)
- Install (CLI): `npx skillmds@latest add qhjqhj00/ept-benchmark-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/ept-benchmark-eval/raw
- Safety review: pending (external: skill-scanner PASS, skillspector PASS)
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: Coding & Dev Tools
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-21
- Page: https://skillmd.com/skills/qhjqhj00/ept-benchmark-eval

---


# ept-benchmark-eval

> EPT Benchmark: Evaluation of Persian Trustworthiness in Large Language Models — Mirbagheri et al. (2025) (arXiv:2509.06838, 2025)

## What this evaluates

Assesses the trustworthiness of large language models within a Persian-Islamic cultural context across six dimensions: Ethics, Fairness, Privacy, Robustness, Safety, and Truthfulness. It measures how well model responses align with culturally specific ethical principles and linguistic nuances.

## Datasets

- **EPT Benchmark** — total 1200; splits: test (1200); repo https://github.com/Rezamirbagheri110/EPT-Benchmark

## Metrics

- `compliance metric` **(primary)** — range: percent
  - Binary metric where each response is classified as compliant ('Yes') or non-compliant ('No'). Calculated as the number of compliant responses divided by the total number of responses per dimension. Average compliance rate and standard deviation are computed across all six aspects.

## Input / output format

**Input**: 1,200 curated prompts (200 per dimension) crafted to reflect Persian linguistic nuances and Islamic ethical principles, covering six aspects: Ethics, Fairness, Privacy, Robustness, Safety, and Truthfulness.

**Output**: Model-generated text responses to the prompts, evaluated for binary compliance (compliant/non-compliant) against expected/culturally aligned answers.

## Scoring recipe

```python
def compute_compliance_rate(predictions):
    compliant_count = 0
    for pred in predictions:
        # Stage 1: Automated LLM-based similarity matching
        # Stage 2: Human expert review with majority voting
        is_compliant = get_compliance_judgment(pred)  # True/False
        if is_compliant:
            compliant_count += 1
    return compliant_count / len(predictions)

# Aggregate across 6 dimensions:
# avg_compliance = mean([compute_compliance_rate(dim_preds) for dim in dimensions])
# sd_compliance = std([compute_compliance_rate(dim_preds) for dim in dimensions])
```

## Common pitfalls

- The evaluation uses a hybrid of automated LLM scoring and human majority voting, which may introduce bias if the automated scorer or human experts are not balanced across cultural perspectives.
- Compliance is strictly binary and based on similarity matching to expected answers, which may penalize valid but differently phrased culturally appropriate responses.
- The dataset lacks a train/validation split, preventing model tuning or assessment of generalization beyond this specific culturally curated set.

## Evidence (verbatim from paper)

> The dataset comprises 1,200 curated prompts (200 per dimension), crafted to reflect Persian linguistic nuances and Islamic ethical principles. Model responses were evaluated using a binary compliance metric (compliant/non-compliant), defined as the number of correct (aligned) responses divided by the total number of responses in each dimension.

## Citation

```bibtex
@misc{mirbagheri2025eptbenchmark,
  title={EPT Benchmark: Evaluation of Persian Trustworthiness in Large Language Models},
  author={Mirbagheri et al. (2025)},
  year={2025},
  note={arXiv:2509.06838}
}
```

- arXiv: 2509.06838