Netops Eval

netops-eval

An Empirical Study of NetOps Capability of Pre-Trained Large Language Models — Miao et al. (2023) (arXiv:2309.05557, 2023)

What this evaluates

Evaluates pre-trained LLMs' networking operations (NetOps) knowledge and reasoning across five technical sub-domains and two languages. It probes both multiple-choice comprehension and open-ended generation capabilities in a domain-specific context.

Datasets

• NetEval — total 5732; splits: development (5), validation (-1), test (-1)

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of correctly answered questions out of the total evaluated. For multiple-choice questions, it is 1 if the predicted option matches the gold option, else 0. For open-ended questions, it relies on rule-based extraction and exact-match or string similarity against the gold answer.

Input / output format

Input: Question text. For multiple-choice items, the four options (A-D) are provided. For open-ended items, only the question or fill-in-the-blank prompt is provided.

Output: For multiple-choice: a single letter (A, B, C, or D). For open-ended: a generated text string representing the answer or command.

Scoring recipe

correct = 0
total = 0
for pred, gold in zip(predictions, golds):
    if gold_type == 'mcq':
        if pred.strip().upper() == gold.strip().upper():
            correct += 1
    else:
        if extract_and_normalize(pred) == extract_and_normalize(gold):
            correct += 1
    total += 1
return correct / total if total > 0 else 0.0

Common pitfalls

• LLMs may guess on multiple-choice questions without possessing actual NetOps knowledge.
• Open-ended generation requires longer context/output, significantly increasing inference cost and latency.
• The full test set is not publicly released; users must submit requests to access evaluation results.

Evidence (verbatim from paper)

The collected evaluation set includes 5,269 multiple-choice questions. ... achieving a human-level accuracy in NetOps certification exam questions is still one of the fundamental capabilities we expect LLMs to have.

Citation

@misc{miao2023netops,
  title={An Empirical Study of NetOps Capability of Pre-Trained Large Language Models},
  author={Miao et al. (2023)},
  year={2023},
  note={arXiv:2309.05557}
}

• arXiv: 2309.05557