Cost-efficient generative AI summarization for scalable automated essay scoring in educational assessment
Quick Answer
This study introduces a generative AI-assisted summarization framework using GPT-5 variants to enhance automated essay scoring (AES) while addressing transformer input-length limitations.
Quick Take
The GPT-5 mini variant achieved the highest agreement with human ratings, indicating a promising approach for scalable writing assessment, although higher-scoring essays showed reduced summary quality due to complexity.
Key Points
- GPT-5 mini achieved the highest agreement with human ratings in AES evaluations.
- Summarization quality decreases for higher-scoring essays, indicating complexity challenges.
- The framework integrates handcrafted linguistic features with summary representations.
- Evaluation metrics include quadratic weighted kappa and various summary quality metrics.
- Generative AI summarization shows promise for scalable educational assessments.
DeepSignal Analysis
What happened
A study proposed a generative AI-assisted summarization framework using GPT-5 variants to enhance automated essay scoring (AES). The framework addresses transformer input-length limitations, which can lead to information loss in long essays. The GPT-5 mini variant showed the highest agreement with human ratings, while the quality of summaries decreased for more complex, higher-scoring essays.
Key evidence
- The study utilized the ASAP 2.0 dataset to evaluate the performance of three GPT-5 variants: GPT-5, GPT-5 mini, and GPT-5 nano.
- Scoring reliability was measured using quadratic weighted kappa (QWK), while summary quality was assessed through various metrics including lexical overlap and semantic similarity.
- Results indicated that the GPT-5 mini variant achieved the highest agreement with human ratings, suggesting its effectiveness for AES.
Why it matters
This research highlights the potential of generative AI in educational assessment, particularly in improving the reliability of automated essay scoring. By addressing input-length limitations, the proposed framework could facilitate more accurate evaluations of student writing. However, the observed trade-offs between summary quality and essay complexity raise concerns about the preservation of critical writing signals, which are essential for fair assessment.
Paper Resources
Source Excerpt
Automated essay scoring (AES) enables scalable assessment and timely feedback but remains challenged by transformer input-length limitations, which can cause information loss when processing long essays. This study proposes a generative AI-assisted summarization framework to improve long-form essay representation while maintaining scoring reliability. Using the ASAP 2. 0 dataset, we generate controlled-length summaries with three GPT-5 variants (GPT-5, GPT-5 mini, and GPT-5 nano) and use them as
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →Letting the Data Speak: Extracting Keywords from Crowdsourced Collections with AI
The study evaluates three NLP approaches—Named Entity Recognition, Keyword Extraction, and Topic Modelling—using the Their Finest Hour Online Archive to automate keyword extraction from crowdsourced WWII collections. Findings suggest that while NLP methods show promise, no single approach is sufficient, and ethical considerations in automated keyword extraction are crucial for responsible stewardship.