Can LLMs Really Understand Item Difficulty Levels? Implications for Automated Item Generation Using LLMs
Quick Answer
This study evaluates LLMs' ability to predict item difficulty levels in assessments, finding that GPT-4.1 achieved a QWK of 0.578, but was outperformed by ConvBERT at 0.625.
Quick Take
The results indicate struggle with labeling hard items, particularly GPT-5.4, which tends to underestimate difficulty, suggesting caution in using LLMs for targeted item generation.
Key Points
- GPT-4.1 achieved a QWK of 0.578 in predicting item difficulty.
- ConvBERT outperformed LLMs with a QWK of 0.625.
- LLMs struggled particularly with labeling hard items.
- GPT-5.4 tends to underestimate item difficulty levels.
- Findings suggest LLMs may not fully grasp item difficulty.
DeepSignal Analysis
What happened
This study assesses the performance of large language models (LLMs) in predicting item difficulty levels for assessments. GPT-4.1 achieved a quadratic weighted kappa (QWK) of 0.578, while ConvBERT outperformed it with a QWK of 0.625. The findings indicate that LLMs, particularly GPT-5.4, tend to underestimate item difficulty, raising concerns about their reliability for targeted item generation.
Key evidence
- GPT-4.1 achieved a QWK of 0.578 in predicting item difficulty levels, marking its highest accuracy in the study.
- ConvBERT outperformed GPT-4.1 with a QWK of 0.625, indicating better performance in predicting item difficulty.
- The study found that all LLMs struggled with labeling hard items, particularly noting that GPT-5.4 underestimated difficulty levels.
Why it matters
Understanding item difficulty is crucial for effective assessment design. The limitations of LLMs in accurately predicting difficulty levels could impact their use in automated item generation, potentially leading to assessments that do not accurately reflect the intended challenge for test-takers. This raises questions about the reliability of LLMs in educational contexts.
Paper Resources
Source Excerpt
The estimation of item difficulty plays a key role in both formative assessment and large-scale high-stakes summative assessments. This study explores how (LLMs) perform in predicting item difficulty levels using items from a large-scale Reading and Writing test. The study investigated various prompting strategies and parameter settings across multiple LLMs. LLM performance was compared with encoder-only language models and feature-based supervised machine learning models.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.