What are They Thinking? Delineation, Probing and Tracking of Concepts in LLMs
Quick Answer
The paper introduces a method for probing LLMs to detect concepts within their embeddings, enabling monitoring of model 'thoughts.' It demonstrates the creation of linear probes for four concepts across three LLMs, paving the way for scalable concept tracking in future models.
Key Points
- Probes detect the presence of concepts in embeddings.
- Method includes delineating concepts with presence and absence datasets.
- Linear probes are trained to detect concepts across LLM layers.
- Four concepts were tested on three different LLMs.
- Scalable approach allows monitoring of many concepts in new models.
Paper Resources
Article Excerpt
From source RSS / original summaryarXiv:2605. 28823v1 Announce Type: new Abstract: As the influence of expands, it is imperative to gain insight into their decisions. One way to do that is to develop probes that detect the presence or absence of a broad set of concepts within the embeddings computed in an LLM - which is what we might say a model is "thinking" about. Such probes should be low-cost and easily applicable to any LLM, so that monitoring for many concepts is possible during normal operation.
In this paper, we take the first steps towards developing the capability of creating many such probes by defining and executing examples of the key tasks needed: first, the careful delineation of a concept through the creation of a dataset with the concept both present and then absent. Then, the training and testing of a set of linear probes to detect the concept on any layer of an LLM, including an exploration of the complexity of the probe needed.
Finally, we show that such probes can track concepts across larger contexts. This is done with four separate concepts and three different LLMs. When this process is scaled to many more concepts, it will create the ability to easily monitor new models.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.