subjective-qa-eval
SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysis — Pardawala et al. (2024) (arXiv:2410.20651, 2024)
What this evaluates
Evaluates models' ability to classify six subjective linguistic features (Assertive, Cautious, Optimistic, Specific, Clear, Relevant) in financial earnings call question-and-answer transcripts. It probes how well models capture nuanced, tone-based, and domain-specific communication cues beyond factual content.
Datasets
Metrics
weighted F1 score (primary) — range: percent
- Computed per feature by averaging class-level F1 scores weighted by their support (true positive count). The final score is the arithmetic mean of the weighted F1 scores across all six features.
Input / output format
Input: Question-answer pairs extracted from earnings call transcripts.
Output: Classification labels for six subjective features: Assertive, Cautious, Optimistic, Specific, Clear, Relevant.
Scoring recipe
scores = []
for feature in ["Assertive", "Cautious", "Optimistic", "Specific", "Clear", "Relevant"]:
y_true, y_pred = get_labels(feature)
scores.append(f1_score(y_true, y_pred, average="weighted"))
return sum(scores) / len(scores)
Common pitfalls
- Train/val/test split sizes are not disclosed in the benchmarking section, hindering exact reproducibility.
- Models are evaluated independently per feature rather than jointly, which may obscure cross-feature trade-offs.
- Weighted F1 is reported as a percentage, but class distribution per feature is not provided, making it hard to assess bias.
Evidence (verbatim from paper)
For each feature, we observed different models performing better. Due to the independence of our features, we can use each model independently to evaluate a given feature. For Clear, BERT had the highest weighted F1 score of 80.93%. For Optimistic and Assertive, RoBERTa-base had the highest weighted F1 scores of 62.69% and 49.10%, respectively. For Relevant, the LLMs, Llama-3-70b-Chat and Mixtral-8x22B Instruct (141B), outperformed the Pre-trained Language Models (PLMs)... Across all six features, RoBERTa-base had the highest average weighted F1 score of 63.95%.
Citation
@misc{pardawala2024subjectiveqa,
title={SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysis},
author={Pardawala et al. (2024)},
year={2024},
note={arXiv:2410.20651}
}
1---2name: subjective-qa-eval3description: Evaluates models' ability to classify six subjective linguistic features (Assertive, Cautious, Optimistic, Specific, Clear, Relevant) in financial earnings call question-and-answer transcripts. It probes how well models capture nuanced, tone-based, and domain-specific communication cues beyond factual content. Use when the user wants to benchmark on SubjECTive-QA, or asks about evaluating this task. Reports weighted F1 score.4---56# subjective-qa-eval78> SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysis — Pardawala et al. (2024) (arXiv:2410.20651, 2024)910## What this evaluates1112Evaluates models' ability to classify six subjective linguistic features (Assertive, Cautious, Optimistic, Specific, Clear, Relevant) in financial earnings call question-and-answer transcripts. It probes how well models capture nuanced, tone-based, and domain-specific communication cues beyond factual content.1314## Datasets1516- **SubjECTive-QA** — total 49446; splits: train (-1), val (-1), test (-1); repo https://github.com/gtfintechlab/SubjECTive-QA1718## Metrics1920- `weighted F1 score` **(primary)** — range: percent21 - Computed per feature by averaging class-level F1 scores weighted by their support (true positive count). The final score is the arithmetic mean of the weighted F1 scores across all six features.2223## Input / output format2425**Input**: Question-answer pairs extracted from earnings call transcripts.2627**Output**: Classification labels for six subjective features: Assertive, Cautious, Optimistic, Specific, Clear, Relevant.2829## Scoring recipe3031```python32scores = []33for feature in ["Assertive", "Cautious", "Optimistic", "Specific", "Clear", "Relevant"]:34 y_true, y_pred = get_labels(feature)35 scores.append(f1_score(y_true, y_pred, average="weighted"))36return sum(scores) / len(scores)37```3839## Common pitfalls4041- Train/val/test split sizes are not disclosed in the benchmarking section, hindering exact reproducibility.42- Models are evaluated independently per feature rather than jointly, which may obscure cross-feature trade-offs.43- Weighted F1 is reported as a percentage, but class distribution per feature is not provided, making it hard to assess bias.4445## Evidence (verbatim from paper)4647> For each feature, we observed different models performing better. Due to the independence of our features, we can use each model independently to evaluate a given feature. For Clear, BERT had the highest weighted F1 score of 80.93%. For Optimistic and Assertive, RoBERTa-base had the highest weighted F1 scores of 62.69% and 49.10%, respectively. For Relevant, the LLMs, Llama-3-70b-Chat and Mixtral-8x22B Instruct (141B), outperformed the Pre-trained Language Models (PLMs)... Across all six features, RoBERTa-base had the highest average weighted F1 score of 63.95%.4849## Citation5051```bibtex52@misc{pardawala2024subjectiveqa,53 title={SubjECTive-QA: Measuring Subjectivity in Earnings Call Transcripts' QA Through Six-Dimensional Feature Analysis},54 author={Pardawala et al. (2024)},55 year={2024},56 note={arXiv:2410.20651}57}58```5960- arXiv: 2410.20651
qhjqhj00