Peek Engagement Prediction Eval

peek-engagement-prediction-eval

PEEK: A Large Dataset of Learner Engagement with Educational Videos — Bulathwela et al. (2021) (arXiv:2109.03154, 2021)

What this evaluates

This benchmark evaluates a model's ability to predict whether a learner will engage with a specific educational video fragment based on their historical interaction sequence and the fragment's content features. It probes sequential behavior modeling and content-based recommendation in informal, self-directed learning environments.

Datasets

• PEEK — total 290535; splits: train (-1), test (-1); repo https://github.com/sahanbull/PEEK-Dataset

Metrics

• F1-measure (primary) — range: [0, 1]
  • Weighted average of per-learner F1 scores, where each learner's weight corresponds to their total interaction count in the system. Computed from a binary confusion matrix (engagement vs. non-engagement).

Input / output format

Input: Sequential history of lecture fragments (1 to t-1) for a given learner, optionally augmented with content features (e.g., Wikipedia concept sets or transcript embeddings) for similarity-based baselines.

Output: Binary engagement prediction $\hat{e}^{t}{\ell,r{i}} \in {0, 1}$ indicating whether the learner will engage with fragment t.

Scoring recipe

learner_f1s = []
for learner in learners:
    preds = get_predictions(learner)
    golds = get_ground_truth(learner)
    tp = sum(p == 1 and g == 1 for p, g in zip(preds, golds))
    fp = sum(p == 1 and g == 0 for p, g in zip(preds, golds))
    fn = sum(p == 0 and g == 1 for p, g in zip(preds, golds))
    prec = tp / (tp + fp) if (tp + fp) > 0 else 0
    rec = tp / (tp + fn) if (tp + fn) > 0 else 0
    f1 = 2 * prec * rec / (prec + rec) if (prec + rec) > 0 else 0
    learner_f1s.append((f1, activity_score(learner)))
weighted_f1 = sum(f1 * w for f1, w in learner_f1s) / sum(w for _, w in learner_f1s)
return weighted_f1

Common pitfalls

• Temporal leakage: Predicting engagement for fragment t using information from fragment t or later violates the sequential design.
• Data leakage in user-similarity features: The user-based Jaccard similarity matrix must be computed exclusively on the training set to avoid leaking test interactions.
• Unweighted averaging: The paper explicitly weights F1 by learner activity; reporting simple macro or micro F1 will yield different results than the reported benchmark.

Evidence (verbatim from paper)

A sequential experimental design is employed, where $\hat{e}^{t}{\ell,r{i}}$, engagement with fragment $t$ is predicted using fragments $1$ to $t-1$. Since engagement is binary, predictions for each fragment can be assembled into a confusion matrix, from which we compute well-known binary classification metrics such as precision, recall and F1-measure. We focus on these measures as we are more interested in predictive lecture fragments that the learners are likely to engage with. We average these metrics per learner and weight each learner according to their amount of activity in the system. We use F1-measure for model comparison as we are interested in improving both precision and recall.

Citation

@misc{bulathwela2021peek,
  title={PEEK: A Large Dataset of Learner Engagement with Educational Videos},
  author={Bulathwela et al. (2021)},
  year={2021},
  note={arXiv:2109.03154}
}

• arXiv: 2109.03154