SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework
Quick Answer
SEFORA introduces a public corpus of 564 drafts and 8,240 instructor annotations to enhance writing feedback.
Quick Take
The UniMatch framework evaluates -generated feedback, revealing a maximum F1 score of 0.4 across 74 configurations, indicating challenges in aligning AI feedback with instructor priorities.
Key Points
- SEFORA corpus captures real instructor feedback across various college writing genres.
- UniMatch framework segments feedback and scores semantic correspondence based on instructor criteria.
- No LLM configuration achieved an F1 score exceeding 0.4 in the evaluation.
- Models struggle to prioritize feedback that instructors deem important.
- Performance decreases as models generate more feedback units.
Paper Resources
📖 Reader Mode
~2 min readAbstract:Effective writing feedback is among the strongest drivers of student learning, yet producing it at scale is labor-intensive. LLMs offer a natural path to scaling writing support, but two gaps stand in the way: few public corpora capture how instructors actually deliver feedback in real classrooms, and no reliable method measures whether generated feedback aligns with what an instructor would write. We address both. SEFORA is a public corpus pairing instructor inline feedback with assignment prompts, rubrics, scores, and multi-draft revisions across various college writing genres, comprising 564 drafts and 8,240 instructor annotations. UniMatch is a reference-based evaluation framework for open-ended generation: it segments feedback into feedback units, scores their semantic correspondence under instructor-derived criteria, and aligns them via optimal matching to yield interpretable precision, recall, and F1. Across 74 experimental configurations spanning multiple LLMs, no setting exceeds 0.4 F1. UniMatch reveals that models struggle to identify the feedback instructors would prioritize, and performance degrades as models generate more.
| Comments: | Under review for EMNLP 2026 |
| Subjects: | Computation and Language (cs.CL); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.00274 [cs.CL] |
| (or arXiv:2607.00274v1 [cs.CL] for this version) | |
| https://doi.org/10.48550/arXiv.2607.00274 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Shayan Peyghambari Oskoui [view email]
[v1]
Tue, 30 Jun 2026 23:48:15 UTC (510 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.