Routenlp Eval

routenlp-eval

RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization — Guo et al. (2026) (arXiv:2604.23577, 2026)

What this evaluates

Evaluates a closed-loop LLM routing system's ability to dynamically select between a four-tier model portfolio based on task difficulty, balancing inference cost, response quality, and latency. The benchmark probes how well a router can escalate queries to more capable models only when necessary, while using distillation and conformal cascading to maintain performance at lower cost tiers.

Datasets

• EDGAR (NER) — total 10000; splits: train (8200), test (1800)
• EDGAR (Summarization) — total 6600; splits: train (5400), test (1200)
• BANKING77 (Intent Classification)* — total 14600; splits: train (12000), test (2600)
• BANKING77 (Response Generation)* — total 8300; splits: train (6800), test (1500)
• CUAD (Clause Extraction)* — total 5600; splits: train (4600), test (1000)
• CUAD (Risk Assessment)* — total 3900; splits: train (3200), test (700)

Metrics

• F1 — range: [0, 1]
  • Macro-averaged F1 score for entity recognition tasks (NER, Clause Extraction).
• ROUGE-L — range: [0, 1]
  • Longest common subsequence overlap between generated summaries and reference summaries.
• BERTScore — range: [0, 1]
  • F1 score computed using contextual embeddings from a pre-trained BERT model to measure semantic similarity.
• accuracy — range: [0, 1]
  • Proportion of correctly predicted intent labels or risk assessment outcomes.
• Quality Ratio (primary) — range: [0, 1]
  • Ratio of the routing system's task-specific quality metric to the Always-T4 upper bound quality.
• Cost Ratio — range: [0, 1]
  • Ratio of cumulative cascade inference costs to the Always-T4 baseline cost.
• p99 latency — range: other
  • 99th percentile end-to-end response time under simulated production load.
• SLA violation rate — range: percent
  • Fraction of requests exceeding predefined latency or quality service-level agreements.

Input / output format

Input: Task-dependent inputs: text spans for NER and Clause Extraction, document text for Summarization and Risk Assessment, and user utterances for Intent Classification and Response Generation.

Output: Task-specific predictions: entity labels for NER/Clause Extraction, summary text for Summarization, intent labels for Intent Classification, generated responses for Response Generation, and risk scores/labels for Risk Assessment.

Scoring recipe

def compute_metrics(predictions, gold, task):
    if task in ['NER', 'Clause Extraction']:
        return f1_score(gold, predictions, average='macro')
    elif task == 'Summarization':
        return rouge_l_score(gold, predictions)
    elif task == 'Intent Classification':
        return accuracy_score(gold, predictions)
    elif task == 'Response Generation':
        return bertscore_f1(gold, predictions)
    elif task == 'Risk Assessment':
        return accuracy_score(gold, predictions)

def compute_ratios(system_quality, system_cost, t4_quality, t4_cost):
    quality_ratio = system_quality / t4_quality
    cost_ratio = system_cost / t4_cost
    return quality_ratio, cost_ratio

Common pitfalls

• Baselines originally designed for 2-model routing were extended to 4-tier settings; using unadapted 2-tier implementations yields unfair cost/quality comparisons.
• Quality and Cost Ratios are computed relative to the Always-T4 upper bound using cumulative cascade costs, not absolute inference costs.
• Statistical significance must be evaluated using paired bootstrap tests over 5 seeds, not simple mean comparisons.

Evidence (verbatim from paper)

Task-specific quality (F1, ROUGE-L, BERTScore, accuracy); Quality Ratio and Cost Ratio relative to Always-T4 (using cumulative cascade costs); p99 latency under simulated production load; SLA violation rate. All experiments over 5 seeds with paired bootstrap significance tests.

Citation

@misc{guo2026routenlp,
  title={RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization},
  author={Guo et al. (2026)},
  year={2026},
  note={arXiv:2604.23577}
}

• arXiv: 2604.23577