Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?
arXiv cs.CL·Prateek Rajput, Yewei Song, Iyiola E. Olatunji, Jacques Klein, Tegawend\'e F. Bissyand\'e
1d agoQuick Take
The study reveals limitations in LLM personality induction despite improved stability through fine-tuning.
Key Points
- Fine-tuning reduces variance in personality questionnaire responses.
- Accuracy on full Big Five profiles remains near chance.
- Suggests need for scenario-grounded datasets for better personality expression.
📖 Reader Mode
~2 min readAbstract:Can large language models reliably express a human-like personality, or are they merely mimicking surface cues without a stable underlying profile? To investigate this, we induce personality in LLMs by fine-tuning them on the long-form essays, where each essay is associated with a target Big Five personality profile. We then evaluate the stability and fidelity of the induced personality using the IPIP-NEO questionnaire. Specifically, we ask: (i) does post-training (SFT, DPO, ORPO) stabilize questionnaire scores under prompt rephrasings, and (ii) can it induce target Big Five profiles from unguided essays? Our results demonstrate that fine-tuning consistently reduces variance in questionnaire responses across five models, directly mitigating the evaluation fragility reported in pre-trained models. However, this newfound stability reveals a more fundamental limitation: accuracy on the full five-dimensional profile remains near chance, even when single-trait scores improve. This indicates that unguided essays lack the cues needed for faithful personality expression. We therefore argue for scenario-grounded datasets or interactive elicitation that accumulates test-aligned evidence over time.
| Comments: | 14 pages, 8 main pages, 5 figures, 4 main page figures |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.16996 [cs.CL] |
| (or arXiv:2605.16996v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.16996 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Prateek Rajput [view email]
[v1]
Sat, 16 May 2026 13:44:06 UTC (1,647 KB)
— Originally published at arxiv.org
More from arXiv cs.CL
See more →arXiv cs.CL·Leyao Wang, Yanan He, Peng Chen, Asaf Yehudai, Yixin Liu, Rex Ying, Michal Shmueli-Scheuer, Arman Cohan
17h ago FeaturedOriginal
Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
AI Summary
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.