Search-on-Graph-R1: Training Large Language Models to Search Knowledge Graphs with Reinforcement Learning
Quick Answer
This paper shows that Search-on-Graph-R1 (SOGR1) is an 8B model that outperforms existing frontier LLMs in knowledge graph question answering (KGQA) by integrating supervised fine-tuning and reinforcement learning.
Quick Take
It achieves superior results on benchmarks like CWQ without auxiliary modules during inference, demonstrating effective navigation through knowledge graphs with fewer search calls.
Key Points
- SOGR1 integrates supervised fine-tuning and reinforcement learning for enhanced KGQA.
- Achieves best results on CWQ compared to other frozen frontier-.
- Utilizes a live Freebase server for grounded knowledge graph navigation.
- Demonstrates fewer search calls needed to reach answers than its SFT initialization.
- No auxiliary modules or LLM judges are used during inference or training.
DeepSignal Analysis
What happened
Search-on-Graph-R1 (SOGR1) is an 8 billion parameter model that integrates supervised fine-tuning and reinforcement learning to improve knowledge graph question answering (KGQA). It outperforms existing large language models (LLMs) on benchmarks like CWQ without requiring auxiliary modules during inference.
Key evidence
- SOGR1 achieves superior results on benchmarks such as WebQSP, CWQ, and GrailQA, surpassing every frozen frontier-LLM system in comparison.
- The model internalizes navigation through knowledge graphs, using a teacher model that follows a known answer path with a live Search tool.
- SOGR1 learns to reach answers in fewer search calls compared to its supervised fine-tuning initialization, indicating effective reinforcement learning.
Why it matters
The development of SOGR1 highlights a shift towards more efficient models in KGQA, reducing the reliance on costly frontier-scale inference. This could lead to broader adoption of knowledge graph technologies in applications where quick and accurate information retrieval is essential. The model's ability to outperform larger models with fewer resources may influence future research and development in the field.
Paper Resources
Source Excerpt
Knowledge graph question answering (KGQA) requires navigating from topic entities to an answer several relations away. Recent methods prompt a frontier to explore the graph through a retrieval tool, but their reliance on frontier-scale inference makes them costly to deploy. We present Search-on-Graph-R1 (\sogrone{}), which internalizes this navigation into a compact 8B model through supervised fine-tuning (SFT) followed by reinforcement learning (RL). Our central idea is to scaffold a fronti
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.CL
See more →RF-Agent: A Practical Framework for Building Language Agents for RFIC Design
RF-Agent introduces a novel framework for RF circuit design using , creating a unique RF-domain reasoning dataset with over 11,000 samples. The study reveals that domain-specific supervised fine-tuning and semantic retrieval strategies significantly enhance RF reasoning performance, particularly for smaller models.