Bangla Sentiment Eval

bangla-sentiment-eval

Sentiment Classification in Bangla Textual Content: A Comparative Study — Md. Arid Hasan et al. (2020) (arXiv:2011.10106, 2020)

What this evaluates

Evaluates the capability of NLP models to classify sentiment in Bangla text. It compares classical machine learning, CNN, FastText, and transformer-based architectures to determine which model family performs best on low-resource Bangla sentiment tasks.

Datasets

• Multiple publicly available Bangla sentiment datasets — total ?; splits: train (-1), dev (-1), test (-1); repo https://github.com/banglanlp/bangla-sentiment-classification

Metrics

• accuracy (primary) — range: [0, 1]
  • The proportion of correctly predicted sentiment labels out of the total number of instances. Calculated as (number of correct predictions) / (total number of predictions).

Input / output format

Input: Preprocessed Bangla text, converted to bag-of-n-gram vectors (unigram, bigram, tri-gram weighted with TF-IDF) for classical models, or tokenized sequences for CNN/FastText/Transformer models.

Output: A discrete sentiment class label (e.g., positive, negative, neutral).

Scoring recipe

def calculate_accuracy(predictions, gold_labels):
    correct = sum(1 for p, g in zip(predictions, gold_labels) if p == g)
    return correct / len(gold_labels)

Common pitfalls

• Transformers are evaluated by running each experiment 10 times with different random seeds and selecting the single best model based on development set accuracy, rather than averaging results across runs.
• Early stopping is applied during training based on development set accuracy with a patience of 200 epochs, which may cause models to stop training before convergence if the dev set is small or noisy.
• Different models use different pre-trained embeddings (Word2Vec, GloVe, FastText official, or Transformer Toolkit tokenizers), making direct architectural comparisons confounded by embedding quality.

Evidence (verbatim from paper)

To reduce the computation without impacting the performance we used early stopping based on the accuracy on the development set. The patience size in early stopping was set to 200. ... we run each experiment 10 times using different random seeds and select the model that performs the best on the development set.

Citation

@misc{hasan2020sentiment,
  title={Sentiment Classification in Bangla Textual Content: A Comparative Study},
  author={Md. Arid Hasan et al. (2020)},
  year={2020},
  note={arXiv:2011.10106}
}

• arXiv: 2011.10106