Distinguishing Artificial from Authentic: Evaluating LLMs for Detecting LLM-Generated Content
Quick Answer
This study evaluates the ability of large language models (LLMs) to detect their own generated content across various educational tasks.
Quick Take
Findings reveal that detection accuracy varies significantly by task type, with better performance in programming exercises compared to short-answer questions. The research underscores the limitations of relying solely on for identifying AI-generated student work.
Key Points
- LLMs show high detection accuracy for programming tasks but struggle with short-answer questions.
- Prompt design and response length significantly affect detection performance in reflective writing.
- LLMs often misjudge their own outputs as more human-like than authentic student responses.
- Detection effectiveness varies greatly across different educational task types.
- Caution is advised when using LLMs as standalone tools for identifying AI-generated work.
DeepSignal Analysis
What happened
A study evaluated the ability of large language models (LLMs) to detect their own generated content across various educational tasks. The results indicated that detection accuracy is highly dependent on the task type, with programming exercises yielding better results than short-answer questions.
Key evidence
- The study found that LLMs performed better in detecting their outputs in programming tasks compared to reflective writing and short-answer questions.
- Detection accuracy varied significantly based on factors such as prompt design and response length, particularly affecting reflective writing tasks.
- LLMs often misjudged their own outputs as more human-like than actual student responses, especially in short-answer questions.
Why it matters
The findings highlight the limitations of using LLMs as standalone tools for identifying AI-generated student work. While LLMs can be effective in certain contexts, their reliability varies significantly across different educational tasks, raising concerns about their use in academic integrity assessments.
What to watch
Paper Resources
Source Excerpt
As (LLMs) are increasingly used by students to generate natural language responses and program code, there is growing interest in whether LLMs themselves can be used to distinguish AI-generated work from human-authored submissions. In this paper, we investigate the extent to which LLMs can detect their own generated content across multiple educational task types, including programming exercises, reflective writing, and short-answer questions. Using authentic student respons
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →TriAgent: Divergence-Aware Committees for Cost-Efficient Financial Sentiment Analysis
TriAgent introduces a cost-efficient multi-agent system for financial sentiment analysis, combining VADER, FinBERT, and Qwen2.5. It achieves an F1 score of ~0.87 with significant savings of $9.3M/year at a 10M-user scale compared to GPT-4o-mini, while also detecting hallucinations with an AUC of 0.90.