Orchid Eval

orchid-eval

ORCHID: A Chinese Debate Corpus for Target-Independent Stance Detection and Argumentative Dialogue Summarization — Zhao et al. (2024) (arXiv:2410.13667, 2024)

What this evaluates

Evaluates models on target-independent stance detection (3-way classification) and argumentative dialogue summarization (overall and stance-specific). It probes the ability to classify conflicting viewpoints in Chinese debates and generate concise, faithful summaries aligned with specific stances.

Datasets

• OrChiD — total 14091; splits: train (11005), dev (1534), test (1550); repo https://github.com/xiutian/OrChiD

Metrics

• Accuracy (primary) — range: [0, 1]
  • Overall classification accuracy: the proportion of correctly predicted stance labels (pro, con, mixed) out of all test instances.
• F1 — range: [0, 1]
  • Per-class F1 score computed separately for pro, con, and mixed stances to account for class imbalance.
• ROUGE-1 F1 (primary) — range: [0, 1]
  • Unigram overlap F1 score between generated summary and gold reference summary.
• ROUGE-2 F1 — range: [0, 1]
  • Bigram overlap F1 score between generated summary and gold reference summary.
• ROUGE-L F1 — range: [0, 1]
  • Longest common subsequence overlap F1 score between generated summary and gold reference summary.
• Human Evaluation Score — range: [1, 5]
  • Average rating on a 1-5 scale across four aspects: conciseness, fluency, faithfulness, and informativeness.

Input / output format

Input: Stance Detection: target claim text (t) and utterance text (c_i). Summarization: sequence of debate utterances (D).

Output: Stance Detection: discrete label {pro, con, mixed}. Summarization: generated text summary (Y).

Scoring recipe

def score_stance(predictions, gold):
    acc = sum(p == g for p, g in zip(predictions, gold)) / len(gold)
    f1s = []
    for stance in ['pro', 'con', 'mixed']:
        tp = sum(1 for p, g in zip(predictions, gold) if p == stance and g == stance)
        fp = sum(1 for p, g in zip(predictions, gold) if p == stance and g != stance)
        fn = sum(1 for p, g in zip(predictions, gold) if p != stance and g == stance)
        prec = tp / (tp + fp) if (tp + fp) > 0 else 0
        rec = tp / (tp + fn) if (tp + fn) > 0 else 0
        f1s.append(2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0)
    return acc, f1s

def score_summary(predictions, gold):
    import rouge
    scorer = rouge.Rouge()
    scores = scorer.get_scores(predictions, gold, avg=True)
    return scores['rouge-1']['f'], scores['rouge-2']['f'], scores['rouge-l']['f']

Common pitfalls

• Imbalanced stance labels (31%/31%/38%) require per-class F1 reporting rather than relying solely on overall accuracy.
• Long debate inputs often exceed model context windows, necessitating chunking, iterative prompting, or hierarchical architectures.
• Pipeline approaches for stance-specific summarization suffer from error propagation if the initial stance detection step is inaccurate.

Evidence (verbatim from paper)

Following Cheng et al. (2022), we report both overall accuracy and per-class (stance) F1 scores. We choose the well-established ROUGE scores as automatic evaluation metrics and report standard F1 scores of ROUGE-1, ROUGE-2 and ROUGE-L.

Citation

@misc{zhao2024orchid,
  title={ORCHID: A Chinese Debate Corpus for Target-Independent Stance Detection and Argumentative Dialogue Summarization},
  author={Zhao et al. (2024)},
  year={2024},
  note={arXiv:2410.13667}
}

• arXiv: 2410.13667