Soar Rna Eval

soar-rna-eval

Single-Cell Omics Arena: A Benchmark Study for Large Language Models on Cell Type Annotation Using Single-Cell Data — Liu et al. (2024) (arXiv:2412.02915, 2024)

What this evaluates

Evaluates large language models on zero-shot and chain-of-thought cell type annotation tasks using single-cell RNA-seq gene expression profiles. It probes the models' ability to translate structured genomic data into textual descriptions and accurately predict cell type labels without fine-tuning.

Datasets

• SOAR-RNA — total ?; splits: test (-1)

Metrics

• BLEU-1 — range: [0, 100]
  • Standard n-gram overlap score measuring unigram precision between predicted and reference cell type names. Scaled to [0, 100].
• BLEU-2 — range: [0, 100]
  • Standard n-gram overlap score measuring bigram precision between predicted and reference cell type names. Scaled to [0, 100].
• Average BLEU (primary) — range: [0, 100]
  • Arithmetic mean of BLEU-1 and BLEU-2 scores. Scaled to [0, 100].

Input / output format

Input: Textual description of the top k differentially expressed genes (sorted by p-value) and associated cell metadata (e.g., tissue type), formatted as a question: 'Given the following markers [genes], what is the specific cell type in [metadata] corresponding to these markers?'

Output: A single predicted cell type name, parsed from the LLM's response.

Scoring recipe

def compute_bleu(predictions, references):
    bleu_1 = corpus_bleu(predictions, references, weights=(1, 0, 0, 0)) * 100
    bleu_2 = corpus_bleu(predictions, references, weights=(0, 1, 0, 0)) * 100
    avg_bleu = (bleu_1 + bleu_2) / 2
    return {'BLEU-1': bleu_1, 'BLEU-2': bleu_2, 'Average BLEU': avg_bleu}

Common pitfalls

• LLMs may generate verbose chain-of-thought reasoning or extra text instead of just the cell type name, requiring strict parsing rules to extract the final answer.
• BLEU scores are sensitive to exact string matching; minor variations in cell type nomenclature (e.g., 'T cell' vs 'T-cell') can artificially lower scores despite biological correctness.
• The top-k gene selection and p-value thresholds are not explicitly standardized in the prompt template, which may introduce variability in input quality across datasets.

Evidence (verbatim from paper)

Table 1: The BLEU evaluation of cell type annotation results on the SOAR-RNA benchmark using the zero-shot and zero-shot chain-of-thought (CoT) prompting strategies to prompt LLMs respectively.

Citation

@misc{liu2024singlecellomicsarena,
  title={Single-Cell Omics Arena: A Benchmark Study for Large Language Models on Cell Type Annotation Using Single-Cell Data},
  author={Liu et al. (2024)},
  year={2024},
  note={arXiv:2412.02915}
}

• arXiv: 2412.02915