# Routenlp Eval

> Evaluates a closed-loop LLM routing system's ability to dynamically select between a four-tier model portfolio based on task difficulty, balancing inference cost, response quality, and latency. The benchmark probes how well a router can escalate queries to more capable models only when necessary, while using distillation and conformal cascading to maintain performance at lower cost tiers. Use when the user wants to benchmark on EDGAR (NER), EDGAR (Summarization), BANKING77* (Intent Classification), BANKING77* (Response Generation), CUAD* (Clause Extraction), CUAD* (Risk Assessment), or asks about evaluating this task. Reports Quality Ratio.

- Skill: `qhjqhj00/routenlp-eval` (Agent Skill)
- Install (CLI): `npx skillmds add qhjqhj00/routenlp-eval`
- Raw SKILL.md: https://api.skillmd.com/api/skills/qhjqhj00/routenlp-eval/raw
- Safety review: pending
- Works with: Claude Code, Claude.ai, OpenAI Codex
- Category: AI & ML
- Author: qhjqhj00 (https://skillmd.com/u/qhjqhj00)
- Updated: 2026-09-08
- Page: https://skillmd.com/skills/qhjqhj00/routenlp-eval

---


# routenlp-eval

> RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization — Guo et al. (2026) (arXiv:2604.23577, 2026)

## What this evaluates

Evaluates a closed-loop LLM routing system's ability to dynamically select between a four-tier model portfolio based on task difficulty, balancing inference cost, response quality, and latency. The benchmark probes how well a router can escalate queries to more capable models only when necessary, while using distillation and conformal cascading to maintain performance at lower cost tiers.

## Datasets

- **EDGAR (NER)** — total 10000; splits: train (8200), test (1800)
- **EDGAR (Summarization)** — total 6600; splits: train (5400), test (1200)
- **BANKING77* (Intent Classification)** — total 14600; splits: train (12000), test (2600)
- **BANKING77* (Response Generation)** — total 8300; splits: train (6800), test (1500)
- **CUAD* (Clause Extraction)** — total 5600; splits: train (4600), test (1000)
- **CUAD* (Risk Assessment)** — total 3900; splits: train (3200), test (700)

## Metrics

- `F1` — range: [0, 1]
  - Macro-averaged F1 score for entity recognition tasks (NER, Clause Extraction).
- `ROUGE-L` — range: [0, 1]
  - Longest common subsequence overlap between generated summaries and reference summaries.
- `BERTScore` — range: [0, 1]
  - F1 score computed using contextual embeddings from a pre-trained BERT model to measure semantic similarity.
- `accuracy` — range: [0, 1]
  - Proportion of correctly predicted intent labels or risk assessment outcomes.
- `Quality Ratio` **(primary)** — range: [0, 1]
  - Ratio of the routing system's task-specific quality metric to the Always-T4 upper bound quality.
- `Cost Ratio` — range: [0, 1]
  - Ratio of cumulative cascade inference costs to the Always-T4 baseline cost.
- `p99 latency` — range: other
  - 99th percentile end-to-end response time under simulated production load.
- `SLA violation rate` — range: percent
  - Fraction of requests exceeding predefined latency or quality service-level agreements.

## Input / output format

**Input**: Task-dependent inputs: text spans for NER and Clause Extraction, document text for Summarization and Risk Assessment, and user utterances for Intent Classification and Response Generation.

**Output**: Task-specific predictions: entity labels for NER/Clause Extraction, summary text for Summarization, intent labels for Intent Classification, generated responses for Response Generation, and risk scores/labels for Risk Assessment.

## Scoring recipe

```python
def compute_metrics(predictions, gold, task):
    if task in ['NER', 'Clause Extraction']:
        return f1_score(gold, predictions, average='macro')
    elif task == 'Summarization':
        return rouge_l_score(gold, predictions)
    elif task == 'Intent Classification':
        return accuracy_score(gold, predictions)
    elif task == 'Response Generation':
        return bertscore_f1(gold, predictions)
    elif task == 'Risk Assessment':
        return accuracy_score(gold, predictions)

def compute_ratios(system_quality, system_cost, t4_quality, t4_cost):
    quality_ratio = system_quality / t4_quality
    cost_ratio = system_cost / t4_cost
    return quality_ratio, cost_ratio
```

## Common pitfalls

- Baselines originally designed for 2-model routing were extended to 4-tier settings; using unadapted 2-tier implementations yields unfair cost/quality comparisons.
- Quality and Cost Ratios are computed relative to the Always-T4 upper bound using cumulative cascade costs, not absolute inference costs.
- Statistical significance must be evaluated using paired bootstrap tests over 5 seeds, not simple mean comparisons.

## Evidence (verbatim from paper)

> Task-specific quality (F1, ROUGE-L, BERTScore, accuracy); Quality Ratio and Cost Ratio relative to Always-T4 (using cumulative cascade costs); p99 latency under simulated production load; SLA violation rate. All experiments over 5 seeds with paired bootstrap significance tests.

## Citation

```bibtex
@misc{guo2026routenlp,
  title={RouteNLP: Closed-Loop LLM Routing with Conformal Cascading and Distillation Co-Optimization},
  author={Guo et al. (2026)},
  year={2026},
  note={arXiv:2604.23577}
}
```

- arXiv: 2604.23577