Tcmsd Sd Eval

tcmsd-sd-eval

TCM-SD: A Benchmark for Probing Syndrome Differentiation via Natural Language Processing — Mucheng Ren et al. (arXiv:2203.10839, 2022)

What this evaluates

Evaluates a model's ability to perform syndrome differentiation in Traditional Chinese Medicine by classifying clinical records into one of 148 predefined syndromes. It probes the model's capacity to handle domain-specific medical terminology and imbalanced multi-class classification.

Datasets

• TCM-SD — total 54152; splits: dev (-1), test (-1); repo https://github.com/Borororo/ZY-BERT

Metrics

• Macro-F1 (primary) — range: [0, 1]
  • The unweighted mean of the F1-score calculated independently for each of the 148 syndrome classes. It treats all classes equally regardless of support.
• Accuracy — range: [0, 1]
  • The proportion of correctly predicted syndromes out of the total number of instances.

Input / output format

Input: Concatenated text: '[CLS] Chief Complaint [SEP] Medical History [SEP]'

Output: A single predicted syndrome label from a fixed set of 148 candidate labels, derived from the [CLS] token representation.

Scoring recipe

def compute_metrics(preds, golds, num_classes=148):
    acc = sum(p == g for p, g in zip(preds, golds)) / len(golds)
    f1_scores = []
    for c in range(num_classes):
        tp = sum(1 for p, g in zip(preds, golds) if p == c and g == c)
        fp = sum(1 for p, g in zip(preds, golds) if p == c and g != c)
        fn = sum(1 for p, g in zip(preds, golds) if p != c and g == c)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
        f1_scores.append(f1)
    macro_f1 = sum(f1_scores) / num_classes
    return {'accuracy': acc, 'macro_f1': macro_f1}

Common pitfalls

• The dataset is highly imbalanced, so Accuracy can be misleadingly high while Macro-F1 remains low.
• Syndromes are context-dependent and abstract, requiring domain-specific pre-training to map symptoms to structured labels effectively.
• Models must strictly follow the [CLS]...[SEP] token format for input concatenation to function correctly.

Evidence (verbatim from paper)

We selected the multi-class classification task as the primary form of SD to directly compare the performances of the existing models against the TCM-SD dataset, and used the accuracy and Macro-F1 as evaluation metrics. Specifically, the chief complaint and medical history were concatenated as the inputs, i.e. [CLS] Chief Complaint [SEP] Medical History [SEP], where [CLS] and [SEP] are special tokens used for classification and separation. Then the model predicts the target syndromes from 148 candidate labels based on the representation of [CLS] token. However, each syndrome in the TCM-SD dataset should have the same importance. Thus, the Macro-F1 is a more accurate metric to evaluate the performances of the models.

Citation

@misc{ren2022tcmsd,
  title={TCM-SD: A Benchmark for Probing Syndrome Differentiation via Natural Language Processing},
  author={Mucheng Ren et al.},
  year={2022},
  note={arXiv:2203.10839}
}

• arXiv: 2203.10839