Teleqna Eval

teleqna-eval

TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge — Maatouk et al. (2023) (arXiv:2310.15051, 2023)

What this evaluates

Evaluates large language models' domain-specific knowledge in telecommunications, covering general terminology, research concepts, and complex technical standards. It also benchmarks model performance against human telecom professionals under strict no-search conditions.

Datasets

• TeleQnA — total 10000; splits: test (10000); repo https://github.com/netop-team/TeleQnA

Metrics

• accuracy (primary) — range: percent
  • Percentage of questions for which the model selected the option marked as correct in the dataset. Calculated as (number of correct predictions / total number of questions) * 100.

Input / output format

Input: Multiple-choice questions with predefined options. In context-augmented experiments, questions are paired with the top-3 most similar 500-word segments from technical standards documents based on embedding similarity.

Output: The model must select exactly one option from the provided choices for each question.

Scoring recipe

correct = 0
for q, options, gold_idx in dataset:
    pred_idx = model.predict(q, options)
    if pred_idx == gold_idx:
        correct += 1
accuracy = (correct / len(dataset)) * 100

Common pitfalls

• Querying questions in large batches (e.g., B=50) reduces accuracy compared to single questions due to increased topic diversity and context dilution.
• Performance heavily depends on providing external context; without top-3 relevant document segments, accuracy on standards questions drops significantly.
• Human baseline requires strict enforcement of a no-search/no-external-reference policy to ensure a fair comparison.

Evidence (verbatim from paper)

As a performance measure, we define the accuracy as the percentage of questions for which the entity at hand selected the option marked as correct in the dataset.

Citation

@misc{maatouk2023teleqna,
  title={TeleQnA: A Benchmark Dataset to Assess Large Language Models Telecommunications Knowledge},
  author={Maatouk et al. (2023)},
  year={2023},
  note={arXiv:2310.15051}
}

• arXiv: 2310.15051