LLM Score Hierarchical F1

llm-score-hierarchical-f1

Lost in Tokenization: Context as the Key to Unlocking Biomolecular Understanding in Scientific LLMs — Zhuang et al. (2025) (arXiv:2510.23127, 2025)

What this evaluates

Evaluates a model's ability to answer open-ended protein questions accurately and predict Enzyme Commission (EC) numbers hierarchically. It probes semantic understanding of biological knowledge and fine-grained functional classification across multiple taxonomic levels.

Datasets

• Protein QA Tasks — total ?; splits: test (-1); repo https://github.com/opendatalab-raiser/CoKE
• EC Number Prediction — total ?; splits: test (-1); repo https://github.com/opendatalab-raiser/CoKE

Metrics

• LLM-Score (primary) — range: [0, 100]
  • An independent adjudicator LLM (DeepSeek-V3) scores generated answers against ground truth on a 0-100 scale based on factual accuracy. The final metric is the average score across all test samples.
• Hierarchical Micro-F1 (primary) — range: [0, 1]
  • F1-Score computed at each of the four EC number levels by truncating predictions and ground truth to N digits. Aggregated as micro-averaged Precision, Recall, and F1 over the entire test set to handle multi-label classification.

Input / output format

Input: QA: generated answer and ground truth answer embedded in a prompt template. EC: predicted EC number(s) and ground truth EC number(s) for each protein instance.

Output: QA: average numerical score (0-100) extracted from adjudicator LLM responses. EC: micro-averaged hierarchical F1-Score (0-1) computed across four classification levels.

Scoring recipe

def compute_llm_score(predictions, golds):
    scores = []
    for pred, gold in zip(predictions, golds):
        prompt = f'Answer: {pred}\nGround Truth: {gold}\nScore 0-100:'
        score = parse_llm_response(adjudicator_llm(prompt))
        scores.append(score)
    return sum(scores) / len(scores)

def compute_hierarchical_f1(preds, golds):
    tp, fp, fn = 0, 0, 0
    for pred_list, gold_list in zip(preds, golds):
        for level in range(1, 5):
            p_trunc = [p.split('.')[:level] for p in pred_list]
            g_trunc = [g.split('.')[:level] for g in gold_list]
            tp += count_set_matches(p_trunc, g_trunc)
            fp += count_false_positives(p_trunc, g_trunc)
            fn += count_false_negatives(p_trunc, g_trunc)
    prec = tp / (tp + fp)
    rec = tp / (tp + fn)
    return 2 * prec * rec / (prec + rec)

Common pitfalls

• Relying on surface-level lexical overlap metrics like BLEU or ROUGE for open-ended protein QA, which fail to capture semantic accuracy and factual consistency.
• Treating EC number prediction as single-label or requiring exact 4-digit matches, which ignores the hierarchical structure and penalizes correct higher-level functional predictions.

Evidence (verbatim from paper)

For the open-ended protein question–answering task, traditional metrics based on lexical overlap (e.g., BLEU, ROUGE) are inadequate for assessing the semantic accuracy and factual consistency of generated answers. To address this, we adopted an automated evaluation methodology leveraging a LLM as an adjudicator, which we term the LLM-Score.

Citation

@misc{zhuang2025lostintokenization,
  title={Lost in Tokenization: Context as the Key to Unlocking Biomolecular Understanding in Scientific LLMs},
  author={Zhuang et al. (2025)},
  year={2025},
  note={arXiv:2510.23127}
}

• arXiv: 2510.23127