Convkgyarn Quality Eval

convkgyarn-quality-eval

ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA datasets with Large Language Models — Pradeep et al. (2024) (arXiv:2408.05948, 2024)

What this evaluates

Assesses the linguistic quality and conversational realism of LLM-synthesized Knowledge Graph QA turns. Probes fluency, factual relevance, diversity, and grammatical correctness across varied interaction styles and noise augmentations.

Datasets

• ConvKGYarn — total ?; splits: General (ConvKGYarnG) (-1), Related (ConvKGYarnR) (-1)

Metrics

• Fluency, Relevance, Diversity, Grammar & Agreement (primary) — range: [1, 5] | percent
  • Human annotators rate each turn on a 1-5 Likert scale across four dimensions: Fluency, Relevance, Diversity, and Grammar. Agreement is calculated as the mean percentage of all ratings where at least two of three annotators agree.

Input / output format

Input: Synthetically generated conversational question turns (with optional modality, deixis, disfluency, and typo augmentations) derived from structured Wikidata facts.

Output: Per-instance Likert ratings (1-5) for Fluency, Relevance, Diversity, and Grammar, plus inter-annotator agreement percentage.

Scoring recipe

# For each dimension (Fluency, Relevance, Diversity, Grammar):
mean_score = sum(annotator_ratings) / len(annotator_ratings)

# Agreement calculation:
agreeing_turns = sum(1 for turn in dataset if len(set(turn.ratings)) < 3)
agreement_pct = (agreeing_turns / len(dataset)) * 100

Common pitfalls

• Likert scores are reported per dimension rather than as a single aggregate score.
• Inter-annotator agreement is defined as the mean percentage of turns where ≥2 of 3 annotators match, not a standard statistical coefficient like Cohen's kappa.
• This protocol evaluates dataset generation quality, not downstream question-answering accuracy.

Evidence (verbatim from paper)

Table 2: The results from the Single Model Rating of the General (ConvKGYarnG) and Related (ConvKGYarnR) set reflecting Likert scores of 1-5 for Fluency, Relevance, Diversity, and Grammar. Agreement scores represent the mean percentage of all scores where at least two of three annotators agree.

Citation

@misc{pradeep2024convkgyarn,
  title={ConvKGYarn: Spinning Configurable and Scalable Conversational Knowledge Graph QA datasets with Large Language Models},
  author={Pradeep et al. (2024)},
  year={2024},
  note={arXiv:2408.05948}
}

• arXiv: 2408.05948