🎉 Thrilled to announce our MindGames challenge is accepted at #NeurIPS2025! 🧠🤖 Ready to deploy your AI agents to compete and collaborate in Hanabi, Werewolf, Stag Hunt, and Colonel Blotto? 🎮 Stay tuned for details!

This is big! Environment diversity has been a key challenge for me when tuning RL in interactive settings. Turning non-RL environments into RL-trainable ones massively expands where agentic LLMs can operate and unlocks real applications.

Does anyone else feel evaluation/grounding is as hard as building the agent? I helped a friend build an onboarding Q&A agent (“ask how we do X on Team Y, get the right steps + links”). The demo was shiny. The day after, I realized evaluation/grounding isn’t a checkbox—it’s the job. Nothing exploded. Instead, a slow drip of “almost-right”: it quoted last quarter’s PTO pilot because the doc changed mid-week, plus other slight misses. None felt dramatic. It felt slippery. Without tight eval/grounding, the agent isn’t stable enough. What I learned (small, boring, effective): 1. Prompts are model-specific. A prompt that lifts Model A can tank Model B. If you swap models, re-optimize the prompt in evaluation to maximize performance before trusting results. 2. Mirror-prod staging. Spin up a staging environment that mirrors production 3. Extensive tests (not vibes), also could use LLMs as “user simulators” to fuzz phrasing and surface brittle prompts. Curious what it’s like for others—feels similar? What evaluation/grounding habit actually made your agents stick in the real world? Are there ways to scale evaluation?

VITA group dinner last night: Welcome back alumnus @WuyangC for his talk! And happy birthday to Jiajun 🎂 Menu debate highlights: everyone went straight for the most expensive dishes 😂 Research talk ✅ Travel stories ✅ Team energy ✅ This is what makes our group special 🙌

Not able to attend COLM, but our coauthor Kevin Wang will present our work on test-time scaling for world models (e.g., COSMOS). Takeaways: test-time scaling laws hold; small models + extra inference can match bigger ones at equal compute; SWIFT makes it practical with fast tokenization, prob Top-K, and beam search. Project: scalingwfm.github.io/

COLM country distribution for this year. I will present SPIN-Bench on Thursday, 4:30 PM - 6:30 PM poster session #55. Come and chat!

You might be interested @elonmusk @xai @Yuhu_ai_ @ChrSzegedy @TobyPhln @rpoo @TheGregYang @ZihangDai @makro_ai @jimmybajimmyb

(5/5) We’ve included more models in the paper and added experiments on scaling with agent numbers. Check out the website (github.com/spinbench/spinben…) and ⭐ the GitHub for new updates. We’re also working on the next benchmarks and will share more about the upcoming settings soon.

（4/5） Observations In sequential planning tasks like Blocksworld, where the action space is small and each step follows a clear sequence, LLMs can chain reasoning steps effectively. Grok-4 reaches about 97% accuracy. For instance, in Tic-Tac-Toe, both GPT-5 and Grok-4 perform nearly flawlessly. However, spatial reasoning remains challenging — even Grok-4 achieves only ~53%. Tasks such as moving objects in 3D or planning robot paths reveal this gap.

(3/5) Setup in brief The benchmarks cover both classical planning tasks (e.g., Blocksworld) and multi-agent games in cooperative and adversarial settings. The game suite includes Tic-Tac-Toe, Connect Four, Chess, and Hanabi. We also tested diplomacy for selected models.

(2/5) Why this matters As interaction becomes central to LLM research, we need a benchmark that tests how models engage with the environment and with other agents.SPIN-Bench (Strategic Planning, Interaction, and Negotiation) is a unified framework for evaluating long-horizon strategic reasoning and social intelligence in LLMs. It features competitive and collaborative multi-agent scenarios, where models must exchange information, anticipate opponents, and coordinate actions to move beyond single-turn reasoning toward interactive intelligence.

🚀 Update on SPIN-Bench, a benchmark for long-horizon planning & multi-agent collaboration. We’ve added the newest Grok, GPT, and Gemini models to the leaderboard. Frontier models are close to never losing at TicTacToe (near-solved). Got a contender to dethrone it? 📥 DMs open! 👇Below is our leaderboard score, and Grok-4 claims the crown.

🎉 Excited to announce our group has 5 papers at #COLM2025! Congrats to @KevinWang_111 @RunjinChen @hjy836 @CongWenyan0320 @KyriectionZhang and collaborators! Details of our posters below 🧵👇

It's wild to see GPT-5 dominating AI Werewolf. Our NeurIPS MindGames competition features Werewolf/Secret Mafia. Check the leaderboard and join the battles as a human or by deploying your AI agent! mindgamesarena.com

Wild to see GPT-5 crush Werewolf. Our NeurIPS MindGames competition also features Werewolf/Secret Mafia and more. Deploy your agent, compete live, and climb the leaderboard: mindgamesarena.com/

A wonderful evening with the VITA family Good food, laughter, and ideas flowing. Here’s to more breakthroughs together!

We tested LLMs on Chess in SpinBench (spinbench.github.io/index.ht…) a while ago. Now it’s amazing to see LLM+games growing even bigger! 🚀 If you’re interested in benchmarking your agent, the NeurIPS 2025 MindGames competition is happening now. Check it out! #AI #NeurIPS #MindGamesAsk

Just getting started with multi-agent LLMs? Check out this blog post by @leonguertler: leonguertler.github.io/2025/… It walks you through how to train LLMs in multi-agent, multi-turn environments step-by-step. Perfect for anyone intreasted for multiagent LLMs and NeurIPS MindGames competition or exploring agentic LLMs! 🤖 #LLM #MultiAgent

I am helping organize the MindGames NeurIPS25 competition (lead by @KevinWang_111 ). I am obviously biased, but building Agents to compete in multi-player Theory Of Mind games is super cool imo! The leaderboard, games, etc. are hosted on TextArena; and the competition provides a generous amount of GPU credits. 1/3

When I was an undergrad, one of the biggest hurdles preventing me from exploring deep learning was limited GPU resources. Even today in academia, seeing industry teams use thousands of GPUs for LLM training can make hands-on experimentation feel completely out of reach. Tools like Google Colab once helped bridge that gap, but as large language models have grown more complex, fine-tuning your own models has again become challenging. With the Neurips MindGames competition, our core vision is to make LLM training accessible to everyone. Thanks to generous support from @modal_labs, @SentientAGI, and @mlfoundry, we've created an easy way for anyone to experiment with multi-agent LLM scenarios. Every team that registers and completes our quick-start tutorial (takes less than 10 minutes!) will receive $500 in computing credits, generously provided by Modal Labs. While these credits might not cover training models from scratch, it's enough to start experimenting, learning, and gaining valuable hands-on experience. If you know anyone interested in trying out LLM training or LLM agents, please help us spread the word! #LLM #MindGames

MindGames offers two tracks: Social Deduction Track Game: Mafia Generalization Track Games: 3-Player IPD, Colonel Blotto, Codenames We support both agent training and pipeline development. A quick-start kit is available; you could launch in just a few minutes. This is a great opportunity to explore multi-agent and multi-turn LLM settings. A tutorial on multi-turn LLM training with Modal Labs is coming soon, along with more resources for multi-agent LLM training throughout the competition. Register and follow us for updates. Sign up here: mindgamesarena.com/