Conversation Disentanglement Eval

conversation-disentanglement-eval

A Large-Scale Corpus for Conversation Disentanglement — Kummerfeld et al. (2018) (arXiv:1810.11118, 2018)

What this evaluates

Probes a model's ability to reconstruct reply relationships in multi-party, entangled text conversations. It requires identifying which message responds to which, handling simultaneous conversations, and distinguishing reply edges from system or directed messages.

Datasets

• IRC Conversation Disentanglement Corpus — total 77563; splits: test (-1)

Metrics

• F1 score for reply-edge prediction (primary) — range: [0, 1]
  • Standard F1 score computed over predicted vs. gold reply edges in the conversation graph. Precision is the fraction of predicted edges that are correct; recall is the fraction of gold edges recovered. F1 is the harmonic mean of precision and recall.

Input / output format

Input: A chronological sequence of messages from a shared IRC channel, including sender identifiers, timestamps, and message text.

Output: A directed graph where each node represents a message and each directed edge (A → B) indicates that message B is a reply to message A. Connected components define individual conversations.

Scoring recipe

def compute_f1_edges(pred_edges, gold_edges):
    pred_set = set(pred_edges)
    gold_set = set(gold_edges)
    tp = len(pred_set & gold_set)
    fp = len(pred_set - gold_set)
    fn = len(gold_set - pred_set)
    precision = tp / (tp + fp) if (tp + fp) > 0 else 0.0
    recall = tp / (tp + fn) if (tp + fn) > 0 else 0.0
    f1 = 2 * precision * recall / (precision + recall) if (precision + recall) > 0 else 0.0
    return f1

Common pitfalls

• Only ~48% of messages contain explicit directed cues (e.g., @username), so models cannot rely solely on mentions.
• System messages start with '==' but are not all system messages, creating ambiguity in edge prediction.
• Users frequently participate in multiple conversations simultaneously, requiring context-aware disentanglement rather than simple clustering.

Evidence (verbatim from paper)

We label this data with a graph in which messages are nodes and edges indicate that one message is a response to another. Each connected component is a conversation.

Citation

@misc{kummerfeld2018largescale,
  title={A Large-Scale Corpus for Conversation Disentanglement},
  author={Kummerfeld et al. (2018)},
  year={2018},
  note={arXiv:1810.11118}
}

• arXiv: 1810.11118