Vocabulary Projection

Vocabulary Projection methods interpret internal model states by projecting them through the unembedding matrix to obtain a distribution over the vocabulary. The core idea is that the unembedding matrix, which maps the final hidden state to output logits, can serve as a universal decoder for intermediate states throughout the model. By inspecting the top-ranked tokens of the resulting distribution, researchers can directly read off the semantic content encoded in any internal object — such as a residual stream state or an attention head output — in terms of the model's output vocabulary.

zjunlp Updated 1.5k repo stars

File contents

zjunlp/mechanist commit 884a37c422

Frequently asked questions

npx get-skill add zjunlp/vocabulary-projection