Cpsdbench Eval

cpsdbench-eval

CPSDBench: A Large Language Model Evaluation Benchmark and Baseline for Chinese Public Security Domain — Tong et al. (2024) (arXiv:2402.07234, 2024)

What this evaluates

Evaluates LLMs on Chinese public security domain tasks including text classification, information extraction, question answering, and text generation. Probes domain-specific accuracy, reliability, and contextual understanding in high-stakes law enforcement scenarios.

Datasets

• Weibo Sentiment Analysis — total 600; splits: test (-1)
• Rumor Detection — total 400; splits: test (-1)
• Telecommunication Fraud Detection — total 400; splits: test (-1)
• Drug-Related Case Reports — total ?; splits: test (-1)
• Public Security Case Reading Comprehension — total ?; splits: test (-1)
• Public Security Case Summary — total ?; splits: test (-1)

Metrics

• Accuracy — range: [0, 1]
  • Calculated as (TP+TN)/(TP+TN+FP+FN). Represents the proportion of correctly classified instances out of the total.
• Precision — range: [0, 1]
  • Calculated as TP/(TP+FP). Measures the proportion of positive predictions that are actually correct.
• Recall — range: [0, 1]
  • Calculated as TP/(TP+FN). Measures the proportion of actual positives that are correctly identified.
• F1-Score (primary) — range: [0, 1]
  • Harmonic mean of Precision and Recall: 2 * (Precision * Recall) / (Precision + Recall). Standard for classification and IE tasks.
• Hybrid IE Metric — range: [0, 1]
  • A two-stage score combining exact match and fuzzy match. Exact match is 1 if pred==gold else 0. Fuzzy match uses Levenshtein distance; if distance ≤ threshold, score is 1. Final score is a weighted sum of exact and fuzzy scores. High-precision entities (names, amounts) use exact match only.

Input / output format

Input: Chinese text instances from public security scenarios (e.g., social media posts, case reports, fraud messages, legal questions), wrapped in task-specific prompts containing role definition, task description, input specifications, and operational constraints.

Output: Task-dependent outputs: discrete class labels for classification; structured entity/relation tuples for information extraction; natural language yes/no or open-ended answers for question answering; and coherent summary paragraphs for text generation.

Scoring recipe

def compute_metrics(preds, golds, task_type):
    if task_type == 'classification':
        acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
        return acc
    elif task_type == 'ie':
        exact_scores = []
        fuzzy_scores = []
        for p, g in zip(preds, golds):
            exact = 1.0 if p == g else 0.0
            exact_scores.append(exact)
            if exact == 0:
                lev = levenshtein_distance(p, g)
                fuzzy_scores.append(1.0 if lev <= THRESHOLD else 0.0)
            else:
                fuzzy_scores.append(0.0)
        return 0.5 * sum(exact_scores) + 0.5 * sum(fuzzy_scores)
    return None

Common pitfalls

• Exact match vs. semantic equivalence: LLM outputs may differ literally but be semantically correct (e.g., '9:00 am' vs 'around 9:00 am'). The hybrid metric accounts for this via Levenshtein distance.
• High-precision entities (names, amounts) in fraud/IE tasks strictly use exact match only, ignoring fuzzy matches, which can penalize minor phrasing variations.
• Dataset sizes are small (dozens to hundreds) due to commercial API costs, limiting statistical power and generalizability of results.

Evidence (verbatim from paper)

For text classification tasks, we have chosen Accuracy, Precision, Recall, and F1-Score as evaluation metrics. ... For information extraction tasks, the typical metrics are Precision, Recall, and F1-Score. ... we have designed a hybrid evaluation metric. It comprises two steps: firstly, calculating the exact match score between the LLM’s output and the label. Secondly, for predictions that are not exact matches, we calculate the Levenshtein distance... Finally, we obtain a comprehensive score by weighting these two types of scores.

Citation

@misc{tong2024cpsdbench,
  title={CPSDBench: A Large Language Model Evaluation Benchmark and Baseline for Chinese Public Security Domain},
  author={Tong et al. (2024)},
  year={2024},
  note={arXiv:2402.07234}
}

• arXiv: 2402.07234