SMAC-Talk: A Natural Language Extension of the StarCraft Multi-Agent Challenge for Large Language Models
Quick Answer
SMAC-Talk is a novel natural language extension of the StarCraft Multi-Agent Challenge designed for evaluating LLM-based agents in cooperative settings.
Quick Take
It features decentralized control and a communication channel to assess agent coordination and trust, using models from the Qwen3.5 family for benchmarking. This open benchmark aims to advance research in environments.
Key Points
- Introduces SMAC-Talk for evaluating agents in multi-agent environments.
- Features decentralized control and natural language communication for agent coordination.
- Includes deceptive communicator scenarios to test trust and decision-making.
- Benchmarks using four models from the Qwen3.5 family.
- Released as an open benchmark to support cooperative multi-agent research.
Paper Resources
Article Content
From source RSS / original summaryarXiv:2606. 04202v1 Announce Type: new Abstract: As become more widely deployed, they are increasingly expected to work alongside other AI agents rather than operating in isolation. Effective coordination in these settings requires agents to communicate, share information and make decisions under uncertainty. We introduce SMAC-Talk, a natural language extension of the StarCraft Challenge for evaluating LLM-based agents in cooperative multi-agent environments.
The environment has several key features such as decentralized control, partial observability and long-horizon decision making. SMAC-Talk includes a natural language communication channel which is used to probe agent coordination and trust. We use this communication channel to construct different evaluation scenarios, including settings with an embedded deceptive communicator that tries to disrupt and deceive allies through communication alone. We provide three agents for benchmarking using 4 models from the Qwen3.
5 family and study how reasoning structure, memory and model scale affect coordination between agents. We release SMAC-Talk as an open benchmark to support the research community in developing and evaluating LLM agents in cooperative multi-agent settings.
Want this in your inbox every morning?
Daily brief at your local 8am — bilingual EN/中文, free.
More from arXiv cs.AI
See more →HOBA: Hierarchical On-Policy Bidding Agents for Adaptive Online Advertising
HOBA (Hierarchical On-policy Bidding Agents) is a novel hierarchical reinforcement learning framework that enhances online advertising bidding systems by improving adaptability and reducing hyperparameter tuning costs. It utilizes a for hyperparameter inference, a SARSA agent for expert model selection, and a dynamic expert pool for bid execution, achieving a +3.6% increase in target cost during large-scale deployment and outperforming state-of-the-art baselines on AuctionNet.