Promptcblue Eval

promptcblue-eval

PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain — Zhu et al. (2023) (arXiv:2310.14151, 2023)

What this evaluates

Evaluates Chinese large language models on five biomedical NLP tasks: information extraction, text classification, natural language inference, dialogue understanding, and content generation. It probes instruction-following, few-shot in-context learning, and parameter-efficient fine-tuning capabilities in a medical domain context.

Datasets

• PromptCBLUE — total ?; splits: train (-1), test (-1)

Metrics

• Instance-level strict micro-F1 (primary) — range: [0, 1]
  • Computed at the instance level where an instance is a complete extracted piece of information (e.g., entity mention + label/status). A prediction is correct only if all keys of the instance match exactly. Aggregated as micro-F1 across all instances.
• Macro-F1 — range: [0, 1]
  • Standard macro-averaged F1 score computed across all class labels, treating each label equally regardless of frequency.
• Micro-F1 — range: [0, 1]
  • Standard micro-averaged F1 score computed by aggregating contributions of all classes to compute the average metric.
• ROUGE-L — range: [0, 1]
  • Recall-Oriented Understudy for Gisting Evaluation based on the longest common subsequence, measuring the longest matching word sequence between generated and reference text.

Input / output format

Input: Instruction prompt concatenated with k few-shot demonstrations (instruction-response pairs) from the training set, or raw training samples for fine-tuning.

Output: Model-generated text, which is post-processed using regular expressions to extract structured predictions (e.g., entity mentions with labels, class labels, or generated medical reports).

Scoring recipe

def compute_metrics(predictions, gold, task_type):
    if task_type == 'extraction':
        tp = sum(1 for p, g in zip(predictions, gold) if p == g)
        fp = len(predictions) - tp
        fn = len(gold) - tp
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        return 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    elif task_type == 'classification':
        return macro_f1(gold, predictions)
    elif task_type == 'nli':
        return micro_f1(gold, predictions)
    elif task_type == 'generation':
        return rouge_l(gold, predictions)
    return 0

Common pitfalls

• Using standard sequence-level metrics like BLEU or ROUGE for information extraction tasks instead of strict instance-level F1.
• Ignoring the 'non-identical' label in classification tasks (e.g., KUAKE-QIC), which is randomly sampled from all labels and significantly lowers F1 scores.
• Assuming domain-specific pretraining alone improves performance without task-specific fine-tuning or prompt tuning.
• Applying a single PEFT method across all task cohorts, whereas the paper shows optimal methods vary by task type.

Evidence (verbatim from paper)

Since metrics like BLUE or ROUGE Lin (2004) can not properly measure how LLMs perform for some of the PromptCBLUE tasks like medical information extraction tasks, we use post-processing scripts to transform the output sequences to structured data formats. PromptCBLUE adopt the following metrics: Instance-level strict micro-F1 for medical information extraction tasks, IMCS-V2-SR and CHIP-MDCFNPC. Here, an instance means a complete piece of information extracted from the given document. For example, in CMeEE-V2, an instance consists of a entity mention extracted and its predicted entity label. And in IMCS-V2-SR, an instance consists of two keys: the entity mention of a symptom, and its status. We adopt the strict metrics, meaning that the model predicts an instance correctly if and only if it correctly predicts an all the keys of an instance. For medical text classification tasks and the IMCS-V2-DAC tasks, we adopt the macro-F1 score. For medical natural language inference tasks, we adopt the micro-F1 score. For the medical content generation tasks, we adopt ROUGE-L Lin (2004) as the metric.

Citation

@misc{zhu2023promptcblue,
  title={PromptCBLUE: A Chinese Prompt Tuning Benchmark for the Medical Domain},
  author={Zhu et al. (2023)},
  year={2023},
  note={arXiv:2310.14151}
}

• arXiv: 2310.14151