Language models influence which news we see, which job applications get a second look, and what a chatbot answers when asked for medical advice. Yet even experts who build these systems often cannot say why models make decisions, making them hard to trust, audit, and correct. Our paper Cross-Layer Discrete Concept Discovery for Interpreting Language Models introduces a method for reading the inner concepts of language models to better understand them.
Peering Inside the Model
In transformer models, information is spread across multiple layers. While current tools can collapse this spread, they run into another issue: they break a single idea, like 'positive sentiment' in a movie review, into an unknown number of smaller pieces similar in meaning (for example, "great," "awesome," and "superb"). Researchers then have to manually combine them or set arbitrary cutoffs, making the model’s reasoning hard to interpret.
To solve this, our tool CLVQ-VAE acts as a translator for the model's inner reasoning. Inspired by the vector quantization idea in image generation (i.e. sorting complex data into a fixed set of distinct categories), it maps each of the model's internal signals to a clear, single entry in a learned 'dictionary'.
By training our tool to predict the higher-level abstract meaning these signals share from its earlier, surface-level read of the text, it learns that "great," "awesome," and "superb" are all echoes of the same concept (e.g. “praise”) and combines those scattered signals into a clear, unified one.
An Effective Method
We tested our method on four models (BERT, RoBERTa, Llama, and Qwen) across sentiment, toxicity, and news classification. Removing one identified concept (e.g. a cluster of praise words) dropped model accuracy by up to 93%, outperforming previous methods and proving that this concept was the main trigger for the model’s reasoning.
For example, when a glowing review was read as positive, the deciding concept was a tight cluster of praise words such as "superb," "outstanding," and "magnificent"; erasing that single concept left the model unable to tell the review was positive.
To check whether the concepts make sense, we showed four AI judges (GPT-4o-mini, Gemini-Flash, Gemini-Flash lite and Claude haiku) the following: a review, the model's prediction, and the concept each method offered as its explanation, then asked which was most convincing.
CLVQ-VAE was chosen about two-thirds of the time. In our human evaluation, we also gave people nothing but a word cloud of a concept, with no review or label attached, and asked them to guess the model's decision. They were right 78% of the time, against 54% for the usual clustering approach.
Because a concept is just a group of words, it also shows when the model goes wrong. One review praised an actor's impersonations, but the model latched onto a concept of imitation words, such as "parody" and "fake," took them as criticism, and flipped its verdict to negative. Another review was only unenthusiastic, yet the model rested on a concept of faint praise, "fine," "decent," "adequate," and judged it positive. In each case, the concept reveals both the mistake and its cause.
The concepts also respond to context. The word "entertainment" maps to a positive concept in an enthusiastic review but a negative one in a harsh review.
As AI takes on higher-stakes roles, explaining its reasoning in human terms is crucial for catching mistakes, checking for bias, and deciding when a model has earned a user’s trust. CLVQ-VAE is a step toward making a model's reasoning legible after the fact, so the concepts behind its answers can be better inspected, checked, and audited.
Our code is openly available for others to use and build on.
