# Teleqna Eval

> Evaluates large language models' domain-specific knowledge in telecommunications, covering general terminology, research concepts, and complex technical standards. It also benchmarks model performance against human telecom professionals under strict no-search conditions. Use when the user wants to benchmark on TeleQnA, or asks about evaluating this task. Reports accuracy.

- Skill: `qhjqhj00/teleqna-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/teleqna-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/teleqna-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/teleqna-eval

---


# teleqna-eval

> TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge — Maatouk et al. (2023) (arXiv:2310.15051, 2023)

## What this evaluates

Evaluates large language models' domain-specific knowledge in telecommunications, covering general terminology, research concepts, and complex technical standards. It also benchmarks model performance against human telecom professionals under strict no-search conditions.

## Datasets

- **TeleQnA** — total 10000; splits: test (10000); repo https://github.com/netop-team/TeleQnA

## Metrics

- `accuracy` **(primary)** — range: percent
  - Percentage of questions for which the model selected the option marked as correct in the dataset. Calculated as (number of correct predictions / total number of questions) * 100.

## Input / output format

**Input**: Multiple-choice questions with predefined options. In context-augmented experiments, questions are paired with the top-3 most similar 500-word segments from technical standards documents based on embedding similarity.

**Output**: The model must select exactly one option from the provided choices for each question.

## Scoring recipe

```python
correct = 0
for q, options, gold_idx in dataset:
    pred_idx = model.predict(q, options)
    if pred_idx == gold_idx:
        correct += 1
accuracy = (correct / len(dataset)) * 100
```

## Common pitfalls

- Querying questions in large batches (e.g., B=50) reduces accuracy compared to single questions due to increased topic diversity and context dilution.
- Performance heavily depends on providing external context; without top-3 relevant document segments, accuracy on standards questions drops significantly.
- Human baseline requires strict enforcement of a no-search/no-external-reference policy to ensure a fair comparison.

## Evidence (verbatim from paper)

> As a performance measure, we define the accuracy as the percentage of questions for which the entity at hand selected the option marked as correct in the dataset.

## Citation

```bibtex
@misc{maatouk2023teleqna,
  title={TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge},
  author={Maatouk et al. (2023)},
  year={2023},
  note={arXiv:2310.15051}
}
```

- arXiv: 2310.15051