Somd Subtask1 Eval

somd-subtask1-eval

Falcon 7b for Software Mention Detection in Scholarly Documents — Khan et al. (2024) (arXiv:2405.08514, 2024)

What this evaluates

Evaluates the ability of token classification models to identify and categorize software mentions within academic sentences. It probes how well models handle class imbalance, subtoken segmentation, and syntactic complexity in scholarly text.

Datasets

• SOMD (Software Mention Detection in Scholarly Publications) — total ?; splits: train (-1), test (-1); repo https://nfdi4ds.github.io/nslp2024/

Metrics

• F1-Score (primary) — range: [0, 1]
  • Standard F1-score computed on exact token-level matches. Precision and recall are calculated over all tokens, counting only predictions that exactly match the gold IOB2 label.

Input / output format

Input: Individual sentences from scholarly publications.

Output: Token-level labels in IOB2 format, classifying each token as a software mention type (e.g., mention, usage, creation) and software type (e.g., application, package), or 'O' for non-mentions.

Scoring recipe

def compute_f1_exact_match(preds, golds):
    correct = sum(1 for p, g in zip(preds, golds) if p == g)
    total_pred = len(preds)
    total_gold = len(golds)
    precision = correct / total_pred if total_pred > 0 else 0.0
    recall = correct / total_gold if total_gold > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1

Common pitfalls

• Subtoken segmentation in transformer models can cause label misalignment if not handled via Unified or Selective labeling strategies.
• Severe class imbalance towards 'O' tokens requires explicit handling (e.g., weighted loss or adaptive sampling) to avoid models predicting only non-mentions.

Evidence (verbatim from paper)

Central to our evaluation is the F1-Score, focusing on exact matches, which serves as a critical metric to quantify the precision and recall of our models in accurately identifying and classifying software mentions. Adherence to the IOB2 format for our submission files ensures our alignment with the standardized training labels, facilitating direct comparison of our model’s performance against established benchmarks.

Citation

@misc{khan2024somd,
  title={Falcon 7b for Software Mention Detection in Scholarly Documents},
  author={Khan et al. (2024)},
  year={2024},
  note={arXiv:2405.08514}
}

• arXiv: 2405.08514