Taxpraben Eval

taxpraben-eval

TaxPraBen: A Scalable Benchmark for Structured Evaluation of LLMs in Chinese Real-World Tax Practice — Hu et al. (2026) (arXiv:2604.08948, 2026)

What this evaluates

Evaluates LLMs on Chinese real-world tax practice tasks spanning classification, generation, structured prediction, and mixed matching. It probes capabilities across Bloom's taxonomy levels, from factual recall and understanding to complex tax strategy planning and risk prevention.

Datasets

• TaxPraBen — total 7300; splits: test (-1); repo https://github.com/Yating-Chen/TaxPraBen

Metrics

• Overall Average (primary) — range: [0, 1]
  • The arithmetic mean of task-specific metrics (ACC, F1, Macro-F1, BERTScore, BARTScore, or EM Accuracy) computed across 5 independent runs. Ranges from 0 to 1, with higher values indicating better performance.
• Accuracy (ACC) — range: [0, 1]
  • Proportion of correctly predicted discrete labels in classification tasks.
• F1 score — range: [0, 1]
  • Harmonic mean of precision and recall for classification tasks.
• Macro-F1 — range: [0, 1]
  • Unweighted mean of F1 scores across all classes in classification tasks.
• BERTScore — range: [0, 1]
  • Semantic similarity score computed using chinese-xlnet-base embeddings between generated and gold text.
• BARTScore — range: [0, 1]
  • Semantic similarity score computed using bart-large-cnn between generated and gold text.
• Exact Match (EM) Accuracy — range: [0, 1]
  • Binary score indicating whether the model's deterministic output (number or text) exactly matches the gold answer.

Input / output format

Input: JSON object containing fields: Id (data_id), query (prompt), text (context), answer (gold response), and optionally choices (for MCQ tasks).

Output: JSON object containing the model's predicted response, structured according to the task type (e.g., discrete label, free-form text, deterministic number/text, or mixed numerical/textual fields with explanatory rationale).

Scoring recipe

def compute_metric(pred, gold, task_type):
    if task_type == 'classification':
        return acc(gold, pred), f1(gold, pred)
    elif task_type == 'generation':
        return bertscore(gold, pred), bartscore(gold, pred)
    elif task_type == 'structured':
        return em_accuracy(gold, pred)
    elif task_type == 'mixed':
        return (em_accuracy(gold, pred) + bertscore(gold, pred)) / 2

# For each run (1 to 5):
#   metrics = [compute_metric(p, g, t) for p, g, t in zip(preds, golds, types)]
#   run_score = mean(metrics)
# final_score = mean(run_score for 5 runs)

Common pitfalls

• Different task types require different metrics (ACC/F1 for classification, BERTScore/BARTScore for generation, EM for structured prediction); applying a single metric across all tasks will yield incorrect scores.
• The final reported score is an average across 5 independent runs, not a single run; failing to average will overstate variance and reduce reliability.
• For mixed tasks (TaxInspect, TaxPlan), the score averages EM Accuracy and BERTScore; ignoring the rationale component or numerical component will misrepresent performance.

Evidence (verbatim from paper)

In accordance with Xie et al. (2023); Hu et al. (2024), we adopt a unified JSON template to organize task-specific evaluation metrics for the 5 output types in TaxPraBen, as detailed below: (a) Text Classification Tasks: Datasets like TaxTopic, TaxSCQ, and TaxCrime involve discrete label prediction. We use Accuracy (ACC), F1 score, and Macro-F1 to measure performance. (b) Text Generation Tasks: Datasets like TaxRecite, TaxSum, TaxQA, TaxBoard, TaxOpinion, and TaxRisk require free-form or structured textual responses. For these, we apply BERTScore and BARTScore to assess semantic similarity. In TaxRisk and TaxInspect, which generate multiple semantic aspects, we compute the average BERTScore. (c) Structured Prediction Tasks: Datasets like TaxRead, TaxCalc, and TaxMCQ require deterministic outputs in fixed formats (e.g., numbers or texts). We evaluate them using Exact Match (EM) Accuracy. (d) Mixed Matching Task: TaxInspect and TaxPlan combine absolute matches for numerical/text fields and semantic matches for explanatory rationale, averaging EM Accuracy and BERTScore for the final score. To ensure fair assessment across tasks and data types, we compute the overall average of metrics

Citation

@misc{hu2026taxpraben,
  title={TaxPraBen: A Scalable Benchmark for Structured Evaluation of LLMs in Chinese Real-World Tax Practice},
  author={Hu et al. (2026)},
  year={2026},
  note={arXiv:2604.08948}
}

• arXiv: 2604.08948