Evaluation Drift in LLM Personality Induction: Are We Moving the Goalpost?
Quick Take
The study reveals limitations in LLM personality induction despite improved stability through fine-tuning.
Key Points
- Fine-tuning reduces variance in personality questionnaire responses.
- Accuracy on full Big Five profiles remains near chance.
- Suggests need for scenario-grounded datasets for better personality expression.
📖 Reader Mode
~2 min readAbstract:Can large language models reliably express a human-like personality, or are they merely mimicking surface cues without a stable underlying profile? To investigate this, we induce personality in LLMs by fine-tuning them on the long-form essays, where each essay is associated with a target Big Five personality profile. We then evaluate the stability and fidelity of the induced personality using the IPIP-NEO questionnaire. Specifically, we ask: (i) does post-training (SFT, DPO, ORPO) stabilize questionnaire scores under prompt rephrasings, and (ii) can it induce target Big Five profiles from unguided essays? Our results demonstrate that fine-tuning consistently reduces variance in questionnaire responses across five models, directly mitigating the evaluation fragility reported in pre-trained models. However, this newfound stability reveals a more fundamental limitation: accuracy on the full five-dimensional profile remains near chance, even when single-trait scores improve. This indicates that unguided essays lack the cues needed for faithful personality expression. We therefore argue for scenario-grounded datasets or interactive elicitation that accumulates test-aligned evidence over time.
| Comments: | 14 pages, 8 main pages, 5 figures, 4 main page figures |
| Subjects: | Computation and Language (cs.CL) |
| Cite as: | arXiv:2605.16996 [cs.CL] |
| (or arXiv:2605.16996v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2605.16996 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Prateek Rajput [view email]
[v1]
Sat, 16 May 2026 13:44:06 UTC (1,647 KB)
— Originally published at arxiv.org
More from arXiv cs.CL
See more →Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents?
The reliability of LLM judges for evaluating deep research agents is critically assessed using the REFLECT benchmark.