OmniToM: Benchmarking Theory of Mind in LLMs via Explicit Belief Modeling
Quick Take
OmniToM introduces a benchmark for evaluating Theory of Mind in LLMs by requiring explicit belief modeling of narrative actors. It reveals that current models struggle with knowledge-access and representational decisions, impacting their ability to track beliefs effectively. Built from 895 stories and 22,343 labeled propositions, it highlights a significant bottleneck in actor-specific belief tracking.
Key Points
- OmniToM benchmarks Theory of Mind via explicit belief modeling in narratives.
- Models evaluated in two stages: Belief Extraction and Belief Labeling.
- Current LLMs face challenges in knowledge-access and belief representation.
- Benchmark built from 895 stories and 22,343 labeled belief propositions.
- Reveals actor-specific belief-tracking bottleneck in existing models.
Article Content
From source RSS / original summaryarXiv:2605. 26322v1 Announce Type: new Abstract: Theory of Mind (ToM), the ability to infer others' knowledge, intentions, and emotions, is commonly evaluated in large language models (LLMs) using end-point question answering, where performance is judged solely by the final answer to a social reasoning query.
This paradigm obscures whether the model actually constructs the underlying mental-state representations required for robust reasoning, particularly in scenarios involving divergent, evolving, or mistaken beliefs. In order to address this research gap, we introduce OmniToM, a benchmark that directly evaluates these representations by requiring explicit modeling of belief structures for all relevant actors within a narrative.
These structures are composed of belief propositions: minimal statements of what an actor takes to be true about the world or another actor's mental state, allowing knowledge, intentions, emotions, and false beliefs to be analyzed in a common format.
Models are evaluated in two stages: Stage 1: Belief Extraction, which extracts from the story the beliefs relevant to its social dynamics, and Stage 2: Belief Labeling, which assigns each belief a seven-dimensional schema label covering recursive order, truth status, knowledge access, explicitness, content type, mental source, and context.
Built from 895 stories from the existing ToMBench story corpus and augmented with 22,343 labeled belief propositions, OmniToM uses a human-calibrated LLM-assisted annotation pipeline. Across diverse models in zero-shot evaluation, OmniToM reveals an actor-specific belief-tracking bottleneck: current LLMs struggle with the knowledge-access and representational decisions required to transform narrative facts into actors' beliefs and shared mental states.
Reader Mode unavailable (could not extract clean content).
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →The Importance of Out-of-Band Metadata for Safe Autonomous Agents: The Redpanda Agentic Data Plane
The Redpanda Agentic Data Plane (ADP) introduces out-of-band metadata channels to enhance the safety of autonomous AI agents, ensuring secure data access and tamper-proof audit trails. This architecture mitigates risks associated with unpredictable AI behavior by enforcing governance throughout the agent lifecycle, demonstrated in a multi-agent trading system with strict data scoping and approval thresholds.