Beat Minecraft survival from screenshots and keyboard input alone

A Redditor had Codex with GPT-6 Astra play a fresh Minecraft Survival world on Normal through screenshots and simulated keys and mouse, and it went on to kill the Ender Dragon without cheats.

Done withCodex· Nov 3, 2026

What they did

The poster ran a long experiment with OpenAI Codex on GPT-6 Astra in a single-player Survival world on Normal difficulty. The world came from a text seed typed in just before generation, with no map, route or plan given to the agent. Codex saw the game only through screenshots of the Minecraft window, plus whatever the normal interface showed, including the F3 debug screen. It acted through ordinary Windows keyboard and mouse input driven from Python, with no Mineflayer, no memory reading and no API for coordinates or nearby blocks. It was free to write and change its own scripts, study screenshots, look up game mechanics, keep persistent notes and change its plans as it went. The toolkit and its history are public on GitHub.

How it went

The poster says the agent beat the Ender Dragon. It was not real-time play: the run took a long period with many pauses for observation and reasoning. The poster stepped in about twice, both times to loosen rules in AGENTS.md that limited how often it could look at the screen, which action sequences it could use and whether it could edit its own tools. They say they gave no gameplay advice. A time-lapse built from screenshots is online; the full session logs, world save and recordings are still being prepared.

Worth knowing

The poster calls it a preliminary report, not an independently verified first, and the post itself was written by ChatGPT. A commenter pointed out how expensive a run like this is, and no cost figure is given. Commenters also asked whether the game was paused while the agent thought; the poster confirmed Escape was among its allowed keys.

Try it

With Codex· 4 parts to fill in

You are going to play [a PC game] using only what a human player has: screenshots of the game window and ordinary keyboard and mouse input sent from [Python or my input tool]. Do not read game memory, use mods or call any API that exposes hidden state. Start by writing a small toolkit to capture the window and send inputs, then test it. Keep a notes file with your goals, what you have learned and your current plan, and update it as you go. Research game mechanics when you are stuck. Pause the game while you think if that is allowed. Work toward [the goal, e.g. finishing the main boss] and stop to report to me if you are stuck on the same obstacle for [number] attempts.

Square brackets are the parts to fill in. The prompt is ours; the story is theirs.

Where they said it

Related