Lexicon Grounding Eval

lexicon-grounding-eval

Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling — Chengxu Zhuang et al. (2024) (arXiv:2403.14551, 2024)

What this evaluates

Evaluates how well a model learns word meanings and general language modeling performance when trained with lexicon-level contrastive visual grounding. Probes concrete vs. abstract word acquisition, verb relation learning, and next-token prediction accuracy on held-out text.

Datasets

• Word Relatedness — total ?; splits: test (-1)
• Semantic Feature Prediction — total ?; splits: test (-1)
• Context Understanding — total ?; splits: test (-1)
• Lexical Relation Prediction — total ?; splits: test (-1)
• SimVerb-3500 — total ?; splits: test (-1)

Metrics

• Perplexity (primary) — range: other
  • Standard cross-entropy perplexity computed on the held-out set of ungrounded text datasets.
• Word Relatedness / Semantic Feature Prediction / Context Understanding / Lexical Relation Prediction — range: [0, 1]
  • Model similarity scores or prediction accuracy compared against human judgment benchmarks. Evaluated via rank correlation or accuracy.
• Negative Loglikelihood — range: other
  • Averaged negative log-likelihood of each word across all its appearances in the test dataset to measure per-word prediction performance.

Input / output format

Input: Unconditioned text sequences (ungrounded datasets) for evaluation; image-caption pairs for training. Evaluation uses text-only inputs.

Output: Next-token probability distributions or word embeddings for similarity comparison.

Scoring recipe

def compute_perplexity(log_probs, gold_tokens):
    nll = -sum(log_probs) / len(gold_tokens)
    return math.exp(nll)

def compute_word_relatedness(model_scores, human_scores):
    return pearsonr(model_scores, human_scores)

def compute_nll_per_word(word_predictions, word_counts):
    return sum(word_predictions) / sum(word_counts)

Common pitfalls

• Perplexity is evaluated only on held-out ungrounded datasets, not the grounded image-caption data.
• Word relatedness benchmarks compare model similarity to human judgment ranks, not exact match scores.
• SimVerb-3500 specifically tests verb pairs, where static visual grounding shows limited benefit compared to concrete nouns.

Evidence (verbatim from paper)

Furthermore, we find that the LCG algorithm achieves better performance than existing algorithms on general language modeling, measured by perplexity in the held-out set of ungrounded datasets (see Fig.[1]C, the leftmost panel).

Citation

@misc{zhuang2024lexiconcontrastive,
  title={Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling},
  author={Chengxu Zhuang et al. (2024)},
  year={2024},
  note={arXiv:2403.14551}
}

• arXiv: 2403.14551