Stratego - LLM Based Games
Benchmarking LLM reasoning in an imperfect-information strategy game
Automated LLM-versus-LLM Stratego evaluation
How the Stratego benchmark measures model behavior
The automated arena runs repeated games between local Ollama models or Hugging Face models. The runner supports different board sizes and prompt variants, while CSV logs preserve the initial prompt, move sequence, metadata, and result for reproducibility.
The benchmark compares win and draw rates, games that reach the turn limit, invalid-move losses, player-position bias, and the number of completed games. A prompt-improvement step can use recent logs to revise the system prompt for later runs.
Technical focus
Implementation details
- The arena runs configurable LLM-versus-LLM matches across multiple board sizes
- CSV logs preserve prompts, moves, timing, invalid actions, and final outcomes
- Scoring combines game results, turn-limit behavior, invalid-move penalties, and player-position effects
Evaluation framework source
The repository contains the game runner, model adapters, GUI, CSV logger, benchmark commands, scoring code, and prompt-improvement workflow.