Dhoroni Eval

dhoroni-eval

Dhoroni: Exploring Bengali Climate Change and Environmental Views with a Multi-Perspective News Dataset and Natural Language Processing — Azmine Toushik Wasi et al. (2024) (arXiv:2410.17225, 2024)

What this evaluates

Evaluates a model's ability to perform multi-dimensional discourse analysis on Bengali climate news articles. It probes capabilities in stance detection, authenticity verification, political influence identification, and various information extraction tasks related to environmental reporting.

Datasets

• Dhoroni — total ?; splits: train (-1), val (-1), test (-1)

Metrics

• Accuracy — range: [0, 1]
  • The proportion of correctly classified instances out of the total number of instances.
• Precision — range: [0, 1]
  • The ratio of true positive predictions to the total number of positive predictions (TP / (TP + FP)).
• Recall — range: [0, 1]
  • The ratio of true positive predictions to the total number of actual positives (TP / (TP + FN)).
• F1 Score (primary) — range: [0, 1]
  • The harmonic mean of precision and recall, calculated as 2 * (Precision * Recall) / (Precision + Recall). Prioritized for evaluation due to class imbalance.

Input / output format

Input: Bengali-language news article text.

Output: Classification label for one of ten predefined discourse dimensions (e.g., stance, authenticity, political influence, scientific data usage, etc.).

Scoring recipe

def compute_metrics(y_true, y_pred):
    acc = sum(1 for t, p in zip(y_true, y_pred) if t == p) / len(y_true)
    tp = sum(1 for t, p in zip(y_true, y_pred) if t == p)
    fp = sum(1 for t, p in zip(y_true, y_pred) if t != p)
    fn = sum(1 for t, p in zip(y_true, y_pred) if t != p)
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    return acc, prec, rec, f1

Common pitfalls

• Accuracy is heavily skewed by class imbalance, making it a misleading primary indicator of performance for minority classes.
• Significant train-validation performance gap indicates overfitting, as models learn training data well but fail to generalize.
• Task definitions for complex dimensions (e.g., Climate/Env. Topics, Authority Involvement) are ambiguous, leading to near-zero model performance.

Evidence (verbatim from paper)

Across the tasks, we can see a clear pattern: accuracy tends to be higher, but the more insightful metrics—precision, recall, and F1 score—reveal deeper issues, particularly with identifying specific classes or more complicated information.

Citation

@misc{wasi2024dhoroni,
  title={Dhoroni: Exploring Bengali Climate Change and Environmental Views with a Multi-Perspective News Dataset and Natural Language Processing},
  author={Azmine Toushik Wasi et al. (2024)},
  year={2024},
  note={arXiv:2410.17225}
}

• arXiv: 2410.17225