Ept Benchmark Eval

ept-benchmark-eval

EPT Benchmark: Evaluation of Persian Trustworthiness in Large Language Models — Mirbagheri et al. (2025) (arXiv:2509.06838, 2025)

What this evaluates

Assesses the trustworthiness of large language models within a Persian-Islamic cultural context across six dimensions: Ethics, Fairness, Privacy, Robustness, Safety, and Truthfulness. It measures how well model responses align with culturally specific ethical principles and linguistic nuances.

Datasets

• EPT Benchmark — total 1200; splits: test (1200); repo https://github.com/Rezamirbagheri110/EPT-Benchmark

Metrics

• compliance metric (primary) — range: percent
  • Binary metric where each response is classified as compliant ('Yes') or non-compliant ('No'). Calculated as the number of compliant responses divided by the total number of responses per dimension. Average compliance rate and standard deviation are computed across all six aspects.

Input / output format

Input: 1,200 curated prompts (200 per dimension) crafted to reflect Persian linguistic nuances and Islamic ethical principles, covering six aspects: Ethics, Fairness, Privacy, Robustness, Safety, and Truthfulness.

Output: Model-generated text responses to the prompts, evaluated for binary compliance (compliant/non-compliant) against expected/culturally aligned answers.

Scoring recipe

def compute_compliance_rate(predictions):
    compliant_count = 0
    for pred in predictions:
        # Stage 1: Automated LLM-based similarity matching
        # Stage 2: Human expert review with majority voting
        is_compliant = get_compliance_judgment(pred)  # True/False
        if is_compliant:
            compliant_count += 1
    return compliant_count / len(predictions)

# Aggregate across 6 dimensions:
# avg_compliance = mean([compute_compliance_rate(dim_preds) for dim in dimensions])
# sd_compliance = std([compute_compliance_rate(dim_preds) for dim in dimensions])

Common pitfalls

• The evaluation uses a hybrid of automated LLM scoring and human majority voting, which may introduce bias if the automated scorer or human experts are not balanced across cultural perspectives.
• Compliance is strictly binary and based on similarity matching to expected answers, which may penalize valid but differently phrased culturally appropriate responses.
• The dataset lacks a train/validation split, preventing model tuning or assessment of generalization beyond this specific culturally curated set.

Evidence (verbatim from paper)

The dataset comprises 1,200 curated prompts (200 per dimension), crafted to reflect Persian linguistic nuances and Islamic ethical principles. Model responses were evaluated using a binary compliance metric (compliant/non-compliant), defined as the number of correct (aligned) responses divided by the total number of responses in each dimension.

Citation

@misc{mirbagheri2025eptbenchmark,
  title={EPT Benchmark: Evaluation of Persian Trustworthiness in Large Language Models},
  author={Mirbagheri et al. (2025)},
  year={2025},
  note={arXiv:2509.06838}
}

• arXiv: 2509.06838