Teleoracle Eval

teleoracle-eval

TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network — Alabbasi et al. (2024) (arXiv:2411.02617, 2024)

What this evaluates

Evaluates domain-specific question answering and retrieval-augmented generation capabilities in telecommunications. Probes a model's ability to accurately answer multiple-choice questions about 3GPP standards and adhere to retrieved context without relying on generalized prior knowledge.

Datasets

• TeleQnA — total 12000; splits: train (10000), test (2000)

Metrics

• Accuracy (primary) — range: percent
  • Calculated as the percentage of correctly answered questions out of the total test set. (Correct Predictions / Total Questions) * 100.
• Faithfulness — range: percent
  • Measures the extent to which the model’s response is informed by the retrieved context. Higher scores indicate stronger alignment between the context and the generated output.

Input / output format

Input: A multiple-choice question (with up to 5 options) accompanied by retrieved context chunks (top 15 from hybrid search + reranker, extended via SelfExtend to fill the context window).

Output: A single selected answer option (1–5) corresponding to the correct choice.

Scoring recipe

def compute_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return (correct / len(gold_labels)) * 100

def compute_faithfulness(predictions, retrieved_contexts):
    # Measures alignment between response and retrieved context [24]
    # Returns percentage score based on external faithfulness evaluator
    pass

Common pitfalls

• Larger general-purpose models often misinterpret specialized telecom terminology by relying on generalized prior knowledge instead of the retrieved context.
• Models may fail to adhere to strict multiple-choice output formats, requiring careful prompting or fine-tuning to extract the correct option.
• Semantic chunking creates variable-length chunks, necessitating a secondary retrieval step to fill the context window and avoid information loss.

Evidence (verbatim from paper)

We use the Faithfulness metric to quantify this effect[[24]]. This metric measures the extent to which the model’s response is informed by the retrieved context. Higher faithfulness indicates stronger alignment between the context and the generated output.

Citation

@misc{alabbasi2024teleoracle,
  title={TeleOracle: Fine-Tuned Retrieval-Augmented Generation with Long-Context Support for Network},
  author={Alabbasi et al. (2024)},
  year={2024},
  note={arXiv:2411.02617}
}

• arXiv: 2411.02617