Chess-Reachy: Autonomous Robotic Chess Gameplay through Large Language Models and Model Context Protocol

Physical AI has rapidly become one of the most challenging but exciting frontiers in artificial intelligence, where AI systems must perceive, reason, and act within the real world. This thesis presents an application that integrates Large Language Models, the Model Context Protocol, and the Reachy Mini desktop humanoid robot to enable autonomous chess games and human-robot interaction. The system is built around a custom MCP server developed using FastMCP, exposing a set of nine tools that allow an AI host such as Claude to control the robot's physical capabilities, including speech, expressions, audio perception, and visual analysis. The core challenge addressed is chessboard understanding: given a photograph of a chess game and the previous board state in ASCII or FEN notation, the system must identify the last move played and return it in Standard Algebraic Notation. To support this, a dataset of approximately 9.800 labeled chessboard images was constructed from 3D rendered sources and used to fine-tune two vision-language models, Qwen3-VL-8B and Gemma-3-4B, using Supervised Fine-Tuning with LoRA adapters and NF4 quantization. Experimental results show that while training converges healthily, exact-match accuracy on 3D rendered images remains limited, with Qwen achieving the best exact-match rate of 8,55\% and Gemma 7,7\%. Despite this, the full application pipeline successfully demonstrates expressive human-robot interaction, conversational capabilities, and correct move identification on 2D chessboard inputs, confirming the viability of LLM-driven Physical AI for complex real-world tasks.