FirstResearch: Auditable Question Formation for LLM Scientific Discovery Agents
Quick Answer
FirstResearch introduces a structured Research Question Certificate for LLMs, enhancing auditability in scientific question formation.
Quick Take
FirstResearch introduces a structured Research Question Certificate for LLMs, enhancing auditability in scientific question formation. It outperforms existing baselines with a score of 4.86/5 compared to 4.38/5, demonstrating that explicit derivation constraints improve the quality of generated scientific questions.
Key Points
- FirstResearch records definitions, assumptions, models, and hypotheses in its Research Question Certificate.
- Achieved a score of 4.86/5 under DeepSeek, outperforming strong baselines.
- Explicit derivation constraints are shown to enhance the auditability of LLM-generated questions.
- Results are preliminary, using LLM judges instead of human experts.
- Code and reproduction scripts are publicly available for further research.
Paper Resources
📖 Reader Mode
~2 min readAbstract:LLM systems for scientific discovery increasingly assist with ideation, literature synthesis, experiment planning, and report generation, but the first research question they propose can remain difficult to audit: it may sound plausible without exposing the mechanism, falsifier, or assumption that a scientist should inspect. We introduce FirstResearch, a first-principles research-question formation framework for scientific LLM agents whose core artifact is a structured Research Question Certificate. The certificate records primitive definitions, assumptions, a mechanism model, a tension or contradiction, a falsifiable hypothesis, a minimal decisive test, and a failure update rule, making the proposed question inspectable before downstream execution. On ten LLM-agent research topics, FirstResearch outperforms controlled prompt-level baselines inspired by AI co-scientist, Agent Laboratory, and AI Scientist-v2 under a primary DeepSeek-blind-judge protocol. A Gemini-2.5-Flash independent-judge rescore of the same 40 baseline packages preserves the system-level ranking, with FirstResearch scoring 4.86/5 versus 4.38/5 for the strongest baseline and Pearson agreement of 0.865 on average score. A one-repeat ablation checkpoint further suggests that the certificate-centered core is the strongest component: certificate-only scoring reaches 4.90/5 under DeepSeek and 4.88/5 under Gemini, while removing certificates drops below 1/5 under both judges. These results are preliminary and use LLM judges rather than human domain experts, but they support a narrow scientific-discovery claim: explicit derivation constraints are a promising mechanism for making LLM-generated scientific questions more auditable. Code, prompts, saved outputs, and reproduction scripts are available at this https URL.
| Subjects: | Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2607.05682 [cs.AI] |
| (or arXiv:2607.05682v1 [cs.AI] for this version) | |
| https://doi.org/10.48550/arXiv.2607.05682 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Yufeng Wang [view email]
[v1]
Mon, 6 Jul 2026 22:52:07 UTC (12 KB)
— Originally published at arxiv.org
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →Onnes: A Physics-Grounded LLM Simulator for Cryogenic Fault Diagnosis in Quantum Computing Infrastructure
Onnes is a physics-grounded digital twin simulator for dilution refrigerators, enhancing cryogenic fault diagnosis in quantum computing. It achieves a classification accuracy of 99.0% using few-shot demonstrations, matching a supervised ML classifier, while maintaining a low false alarm rate of 6.4% on real hardware.