GitHubWhat they did
u/Wonderful-Ad-8579 wanted agents that could build, test and improve game mods by actually playing, without taking over the desktop. Claude Code planned the project and wrote most of it: a Windows DLL that runs inside the game for input and frame capture, and a TypeScript MCP server on top with roughly 100 tools for keyboard, mouse, gamepad and screen. Games run in the background, several at once, one agent each. Development then ran as a loop: each Claude agent got one game and one problem, such as a car stalling at a junction, played through the MCP, found where the tools went wrong, changed the code and measured the same route before and after. Claude Code reviewed each change and ran the tests before it was published. Agents also wrote the per-game profiles that say where the minimap is and what a waypoint looks like. Codex wrote some of the code too.
How it went
The video shows Claude driving to a waypoint in GTA IV, flying and landing a helicopter in GTA V, and walking to a marker, in real time and unedited, plus scenes from Detroit: Become Human, TABS, Halo Wars 2 and Cities: Skylines. The poster says Claude learned the helicopter in under 30 minutes with no human help. It still struggles indoors without a minimap, took a while to find a door in Morrowind, and aiming in shooters is early. Full playthroughs are not there yet.
Worth knowing
The model does not steer frame by frame. It calls a tool like drive to waypoint and a local loop reads the minimap and steers, with Windows OCR and template matching reading menus and HUD text so few frames go to the model. Windows only, single-player and offline games only, nothing with anti-cheat. The poster found another model handled driving slightly better than Claude in their own comparisons.