From Text to Parameters: Predicting Item Parameters from Embedding Regularization with Reliability and Design Ceilings
Quick Answer
This study introduces a framework for predicting item parameters from text embeddings, achieving a predictive R squared of 0.53 for item difficulty in mathematics, while highlighting the limitations in predicting discrimination and pseudo guessing parameters.
Quick Take
This study introduces a framework for predicting item parameters from text embeddings, achieving a predictive R squared of 0.53 for item difficulty in mathematics, while highlighting the limitations in predicting discrimination and pseudo guessing parameters. The findings emphasize the importance of repeated cross-validation to avoid inflated accuracy in calibration applications.
Key Points
- Item difficulty is highly predictable from text embeddings with R squared = 0.53.
- Discrimination and pseudo guessing parameters show lower predictability from text features.
- Reliability ceiling for pseudo guessing is near zero, making it an unviable target.
- Embedding-based regression matches leaderboard RMSE, despite explaining minimal variance.
- Single train-test splits can inflate accuracy by 0.1 to 0.15 in R squared.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Newly developed items must ordinarily be field tested before their psychometric properties are known, creating a cold start problem for item calibration. Predicting item parameters from features is a long standing measurement problem dating back to the Linear Logistic Test Model; modern text embeddings now automate the design matrices traditionally specified by hand. We propose an evaluation framework combining regularized regression on item text embeddings, repeated cross validated R squared reported with its resampling standard deviation, and two performance upper bounds: a reliability ceiling derived from parameter standard errors, and a design ceiling derived from simulation based power calibration. Applying this framework to a mathematics item bank (EEDI) and a medical licensure benchmark (BEA 2024), we find that item difficulty is highly predictable from text (repeated cross validated R squared = 0.53, or about 57% of its reliability ceiling), whereas discrimination and pseudo guessing appear less predictable. However, evaluating these results against our ceilings reveals that this apparent hierarchy stems from target reliability rather than text signal strength: text uniformly recovers 57 to 63% of the reliable variance across difficulty targets, whereas the 3PL pseudo guessing parameter has a reliability ceiling near zero, making it an unviable target at current precision. On BEA, embedding based regression matches leaderboard RMSE despite explaining almost no variance, highlighting the critical need for scale free metrics and explicit ceilings in benchmarking. Finally, we show that a single train and test split can inflate apparent accuracy by 0.1 to 0.15 in R squared, underscoring the necessity of repeated cross validation for calibration support applications and future benchmark construction.
| Subjects: | Computation and Language (cs.CL); Methodology (stat.ME) |
| Cite as: | arXiv:2607.07141 [cs.CL] |
| (or arXiv:2607.07141v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.07141 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Jinsong Chen [view email]
[v1]
Wed, 8 Jul 2026 08:34:46 UTC (52 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Quantifying Prior Dominance in Systems
The study introduces the Normalized Context Utilization (NCU) metric to evaluate Retrieval-Augmented Generation (RAG) systems, revealing that Small Language Models (SLMs) outperform larger models in factual extraction. The findings indicate that traditional scaling laws yield diminishing returns, with a commercial API frequently failing against adversarial evidence due to systemic confidence collapse.