Ccks2017 Cner Eval

ccks2017-cner-eval

Fast and Accurate Recognition of Chinese Clinical Named Entities with Residual Dilated Convolutions — Jiahui Qiu et al. (arXiv:1808.08669, 2018)

What this evaluates

Evaluates a model's ability to identify five types of clinical named entities (diseases, symptoms, exams, treatments, body parts) in Chinese medical texts. It specifically probes character-level sequence labeling performance and the impact of integrating external dictionary features.

Datasets

• CCKS-2017 Task 2 — total 10024; splits: train (7906), test (2118); repo http://www.ccks2017.com/en/index.php/sharedtask/

Metrics

• F1-score (primary) — range: percent
  • Harmonic mean of precision and recall: F1 = 2 * (Precision * Recall) / (Precision + Recall). Precision = TP / (TP + FP), Recall = TP / (TP + FN).

Input / output format

Input: Chinese clinical sentences or clauses (split by commas), processed at the character level, optionally augmented with dictionary feature embeddings.

Output: Sequence of BIO-style entity tags corresponding to each character in the input sentence.

Scoring recipe

def compute_ner_f1(pred_tags, gold_tags):
    def get_spans(tags):
        spans = []
        start = None
        for i, tag in enumerate(tags):
            if tag.startswith('B-'):
                start = (i, tag[2:])
            elif start and (tag == 'O' or tag.startswith('I-') and not tag.startswith('I-' + start[1])):
                spans.append((start[0], i, start[1]))
                start = None
        return set(spans)
    pred_spans = get_spans(pred_tags)
    gold_spans = get_spans(gold_tags)
    tp = len(pred_spans & gold_spans)
    fp = len(pred_spans - gold_spans)
    fn = len(gold_spans - pred_spans)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    return 2 * prec * rec / (prec + rec) * 100

Common pitfalls

• Sentences are split by commas, which may create non-standard boundaries for entity recognition compared to standard NER datasets.
• Character-level labeling avoids word segmentation errors but requires careful handling of rare characters and dictionary feature integration.
• Training speed comparisons are made against Bi-LSTM-CRF, but convergence time varies significantly by hardware and implementation details.

Evidence (verbatim from paper)

This dataset contains 1,596 annotated instances (10,024 sentences) with five types of clinical named entities, including diseases, symptoms, exams, treatments and body parts. ... In the following experiments, widely-used performance measures such as precision, recall, and F1-score are used to evaluate the methods.

Citation

@misc{qiu2018fast,
  title={Fast and Accurate Recognition of Chinese Clinical Named Entities with Residual Dilated Convolutions},
  author={Jiahui Qiu et al.},
  year={2018},
  note={arXiv:1808.08669}
}

• arXiv: 1808.08669