Only Connect Wall Eval

only-connect-wall-eval

Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset — Alavi Naeini et al. (2023) (arXiv:2306.11167, 2023)

What this evaluates

Evaluates creative problem-solving and associative reasoning by testing whether models can correctly group words and identify connections, specifically probing susceptibility to cognitive fixation effects when misleading red herring clues are present.

Datasets

• Only Connect Wall (OCW) — total ?; splits: train (-1), val (-1), test (-1); repo https://github.com/TaatiTeam/OCW

Metrics

• grouping_evaluation (primary) — range: [0, 1]
  • Exact match accuracy of the four predicted word groups to the four ground-truth groups per wall. Evaluated across 16 runs with different seeds and randomized word orderings to account for initialization variance.

Input / output format

Input: Task 1: A list of 16 clue words per wall. Task 2: A list of 16 clue words (or a solved wall without connections) provided in a few-shot prompt with in-context examples.

Output: Task 1: Four predicted groups of words. Task 2: Four predicted connections. Outputs are post-processed to contain exactly 4 predictions, padded with empty strings if fewer are generated.

Scoring recipe

def score_grouping(predictions, gold):
    # predictions and gold are lists of 4 lists of words
    best_match = 0
    for perm in permutations(range(4)):
        if all(set(predictions[i]) == set(gold[perm[i]]) for i in range(4)):
            best_match = 1
            break
    return best_match

Common pitfalls

• Contextual embeddings are sensitive to word order; the paper randomizes word order across 16 runs to mitigate this.
• Out-of-vocabulary clues are handled via mean pooling or BPEmb sub-words, which can affect embedding quality.
• LLM outputs must be strictly padded to 4 groups/connections to match the evaluation format.

Evidence (verbatim from paper)

For the grouping task evaluation (§2.2), we use clustering algorithms on word-embeddings of the sixteen clue words in each wall, to group them into four predicted groups that are subsequently evaluated against the four ground-truth groups for each wall. ... We developed our prompts on the validation set and reported the final performance on the test set.

Citation

@misc{alavinaeini2023onlyconnectwall,
  title={Large Language Models are Fixated by Red Herrings: Exploring Creative Problem Solving and Einstellung Effect using the Only Connect Wall Dataset},
  author={Alavi Naeini et al. (2023)},
  year={2023},
  note={arXiv:2306.11167}
}

• arXiv: 2306.11167