GRPO4Chess: Improving Next Move Generation in Chess Language Models via Reinforcement Learning

In recent decades, chess has served as a cornerstone for artificial intelligence research, from DeepBlue’s 1997 victory over Garry Kasparov to the dominance of engines such as Stockfish and Leela Chess Zero. While these systems rely on brute-force search and carefully engineered evaluations, a new line of research treats chess as a symbolic language to be modeled and generated. However, current chess language models frequently produce illegal Standard Algebraic Notation (SAN), such as invalid moves or false checkmates, which undermines their reliability. This thesis addresses this challenge by training a chess language model with reinforcement learning (RL) using Group Relative Policy Optimization (GRPO). The training corpus consists of elite over-the-board and online games by grandmasters including Magnus Carlsen and Hikaru Nakamura. We design and evaluate reward functions at two levels: syntax-focused rewards, which only assess the formal correctness of SAN, and board-based rewards, which verify whether moves are executable within the actual game state. Our experiments show that GRPO substantially reduces the frequency of illegal moves, with the best-performing models trained under board-based rewards. These models exhibit improved board consistency but also a higher draw rate and reduced win rate. Compared to a supervised fine-tuned (SFT) trained model, GRPO training achieves superior legality but lower playing strength. Applying GRPO on top of SFT further increases legality, though the win rate remains below the base model. Overall, this work demonstrates that RL can significantly improve the syntactic and semantic validity of chess language models, though a trade-off emerges between move legality and competitive performance.