Zero Shot Teacher Feedback Eval

zero-shot-teacher-feedback-eval

Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction — Wang et al. (2023) (arXiv:2306.03090, 2023)

What this evaluates

Evaluates zero-shot performance of LLMs in teacher coaching tasks, including scoring classroom transcripts against observation rubrics, identifying instructional highlights and missed opportunities, and generating actionable pedagogical suggestions.

Datasets

• CLASS & MQI Classroom Transcripts — total 540; splits: test (540); repo https://github.com/rosewang2008/zero-shot-teacher-feedback

Metrics

• Relevance (primary) — range: other
  • Human-rated axis measuring whether the model's response is relevant to the target CLASS or MQI dimension or task. Rated on a 3-point scale.
• Faithfulness — range: other
  • Human-rated axis measuring whether the model's response correctly interprets events in the transcript without hallucination or misinterpretation. Rated on a 3-point scale.
• Insightfulness — range: other
  • Human-rated axis measuring whether the response reveals something beyond the obvious meaning of the transcript line segment. Rated on a 3-point scale.
• Actionability — range: other
  • Human-rated axis measuring whether a pedagogical suggestion can be easily translated into classroom practice. Rated on a 3-point scale.
• Novelty — range: other
  • Human-rated axis measuring whether a suggestion is something the teacher already does in the transcript. Note: 'yes' indicates low novelty (redundant), 'no' indicates high novelty. Rated on a 3-point scale.

Input / output format

Input: Classroom transcript text, optionally accompanied by specific instructions to rate a dimension (e.g., behavior management, instructional dialogue, mathematical explanations, remediation) or to generate highlights, missed opportunities, and pedagogical suggestions.

Output: For scoring tasks: a single integer between 1–7 or 1–3. For highlights, missed opportunities, and suggestions: generated text responses.

Scoring recipe

def evaluate_responses(dataset, model):
    results = []
    for item in dataset:
        response = model.generate(item.transcript)
        if item.task == 'score':
            score = int(response.strip())
            results.append({'task': 'score', 'score': score})
        else:
            ratings = {}
            for axis in ['Relevance', 'Faithfulness', 'Insightfulness', 'Actionability', 'Novelty']:
                ratings[axis] = human_teacher.rate(response, axis, scale=['yes', 'somewhat', 'no'])
            results.append({'task': item.task, 'ratings': ratings})
    return results

Common pitfalls

• Novelty is scored inversely: a 'yes' rating means the suggestion is NOT novel (already present in the transcript), while 'no' means it is novel.
• Faithfulness explicitly checks for hallucination or misinterpretation of transcript events, which is a frequent failure mode for zero-shot models.
• The evaluation relies on only two human teachers rating a small sample (18 segments per code), limiting statistical power and inter-rater reliability analysis.

Evidence (verbatim from paper)

The teachers evaluate the model examples along three axes. One is relevance: Is the model's response relevant to the CLASS or MQI dimension of interest? Two is faithfulness: Does the model's response have the right interpretation of the events that occur in the classroom transcript? We evaluate along this dimension because the model sometimes can hallucinate or misinterpret the events in the transcript when providing examples. Three is insightfulness: Does the model's response reveal something beyond the line segment's obvious meaning in the transcript? ... The teachers evaluate the model suggestions along four axes. One is relevance... Two is faithfulness... Three is actionability: Is the model's suggestion something that the teacher can easily translate into practice... Finally, four is novelty: Is the model suggestion something that the teacher already does in the transcript? ... we ask the teachers to evaluate on a 3-point scale (yes, somewhat, no).

Citation

@misc{wang2023teachercoach,
  title={Is ChatGPT a Good Teacher Coach? Measuring Zero-Shot Performance For Scoring and Providing Actionable Insights on Classroom Instruction},
  author={Wang et al. (2023)},
  year={2023},
  note={arXiv:2306.03090}
}

• arXiv: 2306.03090