Diagnosing Correctness Probes under Self-Judgement Confounding
Quick Answer
This study reveals that language models' self-judgement (SJ) often misaligns with objective correctness (OC), leading to incorrect ranking of responses.
Quick Take
In experiments with four instruction-tuned models, SJ consistently outperformed OC in predicting correct outputs, suggesting that SJ-associated polarity is more transferable across tasks, undermining the reliability of OC semantics.
Key Points
- Conflict cases show OC and SJ can predict opposite output orderings.
- SJ direction transfers above chance across four models with up to 14B parameters.
- OC direction consistently underperforms in expected ordering across conditions.
- Transferability of SJ-associated polarity does not confirm objective correctness semantics.
- Findings persist under various controls, including answer likelihood and sequence length.
DeepSignal Analysis
What happened
The study investigates the discrepancies between self-judgement (SJ) and objective correctness (OC) in language models. It finds that SJ often leads to higher rankings of responses than OC, particularly in high-confidence cases. This misalignment raises questions about the reliability of OC as a measure of correctness.
Key evidence
- The research involved four instruction-tuned models with up to 14 billion parameters, examining the relationship between SJ and OC.
- In cases of high-confidence disagreement, responses endorsed by SJ were frequently ranked above those deemed correct by OC.
- The study found that SJ-associated direction consistently transferred across tasks, while OC-associated direction showed below-chance performance in expected ordering.
Why it matters
This research highlights a critical issue in evaluating language model outputs, as reliance on SJ may mislead assessments of correctness. The findings suggest that the current methods for determining OC may not be robust, potentially affecting applications that depend on accurate output evaluation. Understanding this discrepancy is essential for improving model reliability and trustworthiness.
Paper Resources
Source Excerpt
Hidden-state readouts can predict whether language-model outputs are correct, but objective correctness (OC) usually agrees with the model's own self-judgement (SJ), leaving the decoded signal semantically ambiguous. We construct conflict cases in which OC and SJ predict opposite readout orderings. On high-confidence disagreements, conventional correctness-labelled contrasts often rank incorrect/self-endorsed responses above correct/self-rejected responses, following SJ rather than OC. We estima
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.