LLM AgentsSupervised Fine-TuningReinforcement Learning

The Village at Scale

LLM Agents Playing Werewolf - learning social reasoning through self-play

The Village at Scale combines supervised fine-tuning and reinforcement learning through self-play to train compact agents for strategic reasoning in Werewolf.

From distilled gameplay to self-play

Reinforcement learning can't get started if the model's output can't be parsed. Untrained agents leak their secret roles into public discussion and ignore the environment's required format entirely — so the pipeline runs in two stages, and the order matters.

The first stage teaches shape rather than skill: a small model learns from a larger one's clean games what a correct turn looks like — reason privately, speak publicly, vote in the expected format. Only then does self-play begin, where the sole feedback is whether the game was won. Because each improved checkpoint becomes the opponent in the next round, the agents train against opposition that is improving alongside them.

Reinforcement learning loop from TextArena gameplay and dataset generation through LoRA policy updates to a new vLLM model checkpoint — FigureAn iteration of the self-play loop.

Technical focus

Multi-agent LLM gameplayHidden-role reasoningSupervised fine-tuningGRPOLoRA adaptersvLLM servingSelf-play

Implementation details

A larger model supplies clean gameplay trajectories for supervised fine-tuning of a smaller agent
Self-play matches turn game outcomes into reinforcement-learning signals
GRPO training with LoRA adapters, served through vLLM, so the improved model becomes the opponent in the next round
A streaming Gradio interface renders all six agents, phases, actions, observations, winners, and rewards

Project repository

Explore the source code and project materials on GitHub.

Open on GitHub

← Back to All Projects