banglabook-sentiment-eval
BanglaBook: A Large-scale Bangla Dataset for Sentiment Analysis from Book Reviews — Kabir et al. (2023) (arXiv:2305.06595, 2023)
What this evaluates
Evaluates the ability of models to classify Bangla book reviews into three sentiment categories (Positive, Neutral, Negative). It probes product-specific sentiment analysis in a low-resource language, testing both contextual understanding and robustness to class imbalance and lexical overlap.
Datasets
Metrics
F1-score — range: [0, 1]
- Harmonic mean of precision and recall, calculated per class to balance performance across imbalanced positive/negative samples.
weighted average F1-score (primary) — range: [0, 1]
- Weighted mean of per-class F1-scores, where weights correspond to the number of true instances for each class, providing a single aggregate score for the entire dataset.
Input / output format
Input: Raw Bangla text of a book review.
Output: Categorical sentiment label: 'Positive', 'Neutral', or 'Negative'.
Scoring recipe
from sklearn.metrics import f1_score
# predictions and gold are lists of class labels
f1_macro = f1_score(gold, predictions, average='macro')
f1_weighted = f1_score(gold, predictions, average='weighted')
Common pitfalls
- Class imbalance causes models to heavily favor the 'Positive' class, leading to poor 'Neutral' and 'Negative' recall.
- High lexical overlap between classes (e.g., positive words appearing in negative/neutral reviews) confuses bag-of-words and n-gram features.
- Rating-based labeling may introduce incongruent samples where the text sentiment contradicts the numerical rating, affecting ground truth reliability.
Evidence (verbatim from paper)
We select F1-score and weighted average F1-score to evaluate the models because the dataset has an uneven class distribution. F1-score is the harmonic mean of precision and recall and it helps balance the metric across the imbalanced positive/negative samples (Sokolova et al., 2006). All our experiments are done using scikit-learn, pytorch, and transformers (Vaswani et al., 2017) and run on Google Colaboratory. The training, testing, and validation split of the entire dataset was 70-20-10 with previously unseen samples in the test and validation set.
Citation
@misc{kabir2023banglabook,
title={BanglaBook: A Large-scale Bangla Dataset for Sentiment Analysis from Book Reviews},
author={Kabir et al. (2023)},
year={2023},
note={arXiv:2305.06595}
}
1---2name: banglabook-sentiment-eval3description: Evaluates the ability of models to classify Bangla book reviews into three sentiment categories (Positive, Neutral, Negative). It probes product-specific sentiment analysis in a low-resource language, testing both contextual understanding and robustness to class imbalance and lexical overlap. Use when the user wants to benchmark on BANGLABOOK, or asks about evaluating this task. Reports weighted average F1-score.4---56# banglabook-sentiment-eval78> BanglaBook: A Large-scale Bangla Dataset for Sentiment Analysis from Book Reviews — Kabir et al. (2023) (arXiv:2305.06595, 2023)910## What this evaluates1112Evaluates the ability of models to classify Bangla book reviews into three sentiment categories (Positive, Neutral, Negative). It probes product-specific sentiment analysis in a low-resource language, testing both contextual understanding and robustness to class imbalance and lexical overlap.1314## Datasets1516- **BANGLABOOK** — total 158065; splits: train (-1), val (-1), test (-1); repo https://github.com/mohsinulkabir14/BanglaBook1718## Metrics1920- `F1-score` — range: [0, 1]21 - Harmonic mean of precision and recall, calculated per class to balance performance across imbalanced positive/negative samples.22- `weighted average F1-score` **(primary)** — range: [0, 1]23 - Weighted mean of per-class F1-scores, where weights correspond to the number of true instances for each class, providing a single aggregate score for the entire dataset.2425## Input / output format2627**Input**: Raw Bangla text of a book review.2829**Output**: Categorical sentiment label: 'Positive', 'Neutral', or 'Negative'.3031## Scoring recipe3233```python34from sklearn.metrics import f1_score35# predictions and gold are lists of class labels36f1_macro = f1_score(gold, predictions, average='macro')37f1_weighted = f1_score(gold, predictions, average='weighted')38```3940## Common pitfalls4142- Class imbalance causes models to heavily favor the 'Positive' class, leading to poor 'Neutral' and 'Negative' recall.43- High lexical overlap between classes (e.g., positive words appearing in negative/neutral reviews) confuses bag-of-words and n-gram features.44- Rating-based labeling may introduce incongruent samples where the text sentiment contradicts the numerical rating, affecting ground truth reliability.4546## Evidence (verbatim from paper)4748> We select F1-score and weighted average F1-score to evaluate the models because the dataset has an uneven class distribution. F1-score is the harmonic mean of precision and recall and it helps balance the metric across the imbalanced positive/negative samples (Sokolova et al., 2006). All our experiments are done using scikit-learn, pytorch, and transformers (Vaswani et al., 2017) and run on Google Colaboratory. The training, testing, and validation split of the entire dataset was 70-20-10 with previously unseen samples in the test and validation set.4950## Citation5152```bibtex53@misc{kabir2023banglabook,54 title={BanglaBook: A Large-scale Bangla Dataset for Sentiment Analysis from Book Reviews},55 author={Kabir et al. (2023)},56 year={2023},57 note={arXiv:2305.06595}58}59```6061- arXiv: 2305.06595
qhjqhj00