OpenAI炮轰AI评测"标杆"： 731 道题近三成有缺陷 - AI NEWS
Quick Answer
OpenAI challenges the SWE-Bench Pro benchmark, claiming 30% of its 731 tasks have flaws, undermining its credibility.
Quick Take
The rapid increase in pass rates from 23.3% to 80.3% in just eight months raises concerns about the benchmark's ability to accurately assess AI models' programming skills.
Key Points
- OpenAI identifies 200 flawed tasks, representing 27.4% of Pro's total.
- A second review found 249 flawed tasks, increasing the defect rate to 34.1%.
- Issues include overly strict testing and misleading prompts affecting model evaluations.
- OpenAI calls for a new assessment framework designed by experienced software developers.
- The current benchmark's flaws challenge the credibility of the entire AI evaluation system.
📖 Reader Mode
~1 min readOpenAI发布博文公开挑战行业权威评测基准SWE-Bench Pro，认为在 731 个公开测试任务中，约30%存在评测缺陷。SWE-Bench Pro由Scale AI推出，专门评测大语言模型与AI智能体的编程能力，因高度贴近实际企业级开发且具备极高的防作弊标准，现已成为AI软件工程领域的行业权威基准。
OpenAI在博文中指出一个关键信号：前沿模型在该基准上的通过率仅用 8 个月就从23.3%飙升至80.3%。这一"进步"速度过于异常，OpenAI认为该基准已无法有效评估模型的真实软件开发能力，很可能是因为评测本身存在系统性问题，而非模型能力真的实现了如此飞跃。
两条审查路径交叉验证，近三成任务"不合格"
为验证判断，OpenAI启动了两条并行审查路径。其数据点分析流程标记出 200 个失效任务，占全部 731 个公开任务的27.4%；同时开展的人工标注活动识别出 249 个失效任务，占比34.1%。基于这两条路径的交叉验证，OpenAI预估SWE-Bench Pro约30%的任务存在缺陷，涉及测试过严、提示不充分、测试范围过窄和提示具有误导性四类问题。
OpenAI还披露了一个典型案例：某道题要求将内容转为Markdown时在行首加入 1 个空格，但隐藏测试却要求 2 个空格。这意味着模型完全按照题面说明编写代码，仍然会被判错。这类"隐藏要求与明文说明不一致"的问题，直接导致模型的真实能力被错误评估，也解释了为何通过率会出现不合理飙升。
撤回采用建议，呼吁重建AI评测体系
基于这次分析，OpenAI正式撤回此前对SWE-Bench Pro的采用建议。OpenAI认为，后续需要由资深软件开发者专门为AI评测设计新的基准，而非简单沿用面向人类开发者的测试逻辑。当行业"标杆"本身都可能存在近三成缺陷，整个AI评测体系的公信力便面临拷问。从跑分竞赛回归真实工程能力评估，或许是AI软件工程评测下一步必须跨越的门槛。
— Originally published at news.aibase.com
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from WebSearch (Tavily)
See more →全球AI芯片峰会，9月上海见！
The 2026 Global AI Chip Summit will take place in Shanghai on September 22-23, focusing on the evolving AI chip landscape, including the shift from training to inference, the rise of diverse chip technologies, and the restructuring of industry competition. Notable speakers include experts from leading universities and companies, discussing advancements in AI chip architecture and applications.