Findata Eval

findata-eval

When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP — Ni et al. (2023) (arXiv:2305.14007, 2023)

What this evaluates

Evaluates multi-task and single-task learning models on a curated collection of financial NLP tasks (FinDATA) to assess how task diversity, relatedness, and parameter-efficient architectures impact performance.

Datasets

• FinDATA — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• evaluation metrics in Table 2 (primary) — range: varies by task
  • Task-specific metrics defined in Table 2 of the paper, used to select the best validation checkpoints and report test performance.

Input / output format

Input: Text inputs corresponding to various financial NLP tasks within the FinDATA collection.

Output: Task-specific predictions or labels, evaluated against gold labels using the task-specific metrics.

Scoring recipe

# For each task t in FinDATA:
val_score = compute_metric(predictions, gold, task=t)
if val_score > best_val_score[t]:
    best_val_score[t] = val_score
    best_checkpoint[t] = current_checkpoint
test_score = compute_metric(best_checkpoint[t].predict(test_data), test_gold, task=t)
final_results = average(test_scores across seeds 1-5) with std_dev

Common pitfalls

• MTL checkpoint selection is performed independently per task, not jointly across all tasks.
• Financial PhraseBank is explicitly excluded from the main model comparison due to lacking an official test set.
• All results must be averaged over random seeds 1 to 5 with standard deviations reported.

Evidence (verbatim from paper)

We follow the evaluation metrics in Table 2 to select the best checkpoints and report the test performance. All MTL and STL results are averaged over random seeds from 1 to 5 with standard deviations attached.

Citation

@misc{ni2023mtlfinancialnlp,
  title={When Does Aggregating Multiple Skills with Multi-Task Learning Work? A Case Study in Financial NLP},
  author={Ni et al. (2023)},
  year={2023},
  note={arXiv:2305.14007}
}

• arXiv: 2305.14007