Cfbenchmark Basic Eval

cfbenchmark-basic-eval

CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model — Lei et al. (2023) (arXiv:2311.05812, 2023)

What this evaluates

Evaluates Chinese large language models on financial text processing capabilities, specifically entity recognition, text classification, and content generation within the financial domain. It tests the models' adaptability using zero-shot and few-shot (3 examples) prompting strategies across eight distinct tasks.

Datasets

• CFBenchmark-Basic — total 3917; splits: test (-1); repo https://github.com/TongjiFinLab/CFBenchmark

Metrics

• F1-Score (primary) — range: [0, 1]
  • Standard harmonic mean of precision and recall for entity recognition tasks. Calculated as 2 * (precision * recall) / (precision + recall).
• Score — range: [0, 1]
  • Reported as a normalized evaluation score for classification and generation tasks. The exact calculation method (e.g., LLM-judge, human eval, or automated metric) is not specified in the provided text.

Input / output format

Input: Instruction prompt concatenated with the target financial text. For few-shot evaluation, the prompt includes the target text plus three example input-output pairs.

Output: Model-generated text containing recognized entities, predicted class labels, or generated financial content based on the prompt.

Scoring recipe

def compute_metrics(predictions, golds, task_type):
    if task_type == 'entity_recognition':
        tp = len(set(predictions) & set(golds))
        fp = len(set(predictions) - set(golds))
        fn = len(set(golds) - set(predictions))
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return {'F1-Score': 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0}
    else:
        # Classification and generation scores are reported as normalized values [0, 1]
        # Exact calculation method is not specified in the text
        return {'Score': golds}

Common pitfalls

• Models tend to omit required entities or misclassify non-relevant entities (e.g., personal names, sectors) as target entities.
• Classification models often generate answers outside the predefined category set, even with few-shot prompting.
• Generation models produce generic, repetitive content (e.g., standard risk phrases) lacking text-specific details.

Evidence (verbatim from paper)

In the domain of financial entity recognition, Qwen-Chat-14B emerges as a frontrunner, registering an F1-Score of 0.628. It is closely trailed by ERNIE-Bot-4 and Qwent-14B, who clock in F1-Scores of 0.618 and 0.613, respectively.

Citation

@misc{lei2023cfbenchmark,
  title={CFBenchmark: Chinese Financial Assistant Benchmark for Large Language Model},
  author={Lei et al. (2023)},
  year={2023},
  note={arXiv:2311.05812}
}

• arXiv: 2311.05812