🎮 pokemon-agent (jriyuu-labs build)

AI-powered Pokémon gameplay agent with headless emulation, REST API, and live dashboard. Forked from NousResearch/pokemon-agent (MIT) and packaged as a Docker image for the jriyuu-labs registry.

Let any AI agent — an OpenClaw agent, Hermes, Claude Code, or your own — play Pokémon games autonomously over a clean HTTP API. Runs headlessly on any Linux box (amd64 or arm64). No display, no GUI, no emulator window needed.

┌──────────────────────┐
│   Your AI Agent      │  Any LLM-powered agent
│   (OpenClaw, Hermes, │  makes the decisions
│    Claude, custom)   │
└─────────┬────────────┘
          │ HTTP API
┌─────────▼────────────┐
│   pokemon-agent      │  This container:
│   ┌────────────────┐ │  - Headless emulator (PyBoy)
│   │ Game Server    │ │  - Memory reader
│   │ (FastAPI)      │ │  - Game state parser
│   ├────────────────┤ │  - REST + WebSocket API
│   │ Emulator       │ │  - Live dashboard
│   └────────────────┘ │
└──────────────────────┘

Docker quick start

Image: nidavellir.jriyuu.io/jriyuu-labs/pokemon-agent — multi-arch (latest, plus per-sha and per-arch tags), built on every push to main by Forgejo CI (.forgejo/workflows/build.yml).

You must provide your own ROM file — the image does not include any game ROMs.

# 1. fetch the image
docker pull nidavellir.jriyuu.io/jriyuu-labs/pokemon-agent:latest

# 2. run it — ROM read-only at /roms, persistent data at /data
docker run -d --name pokemon-agent --restart unless-stopped \
  -p 8765:8765 \
  -v $PWD/roms:/roms:ro \
  -v $PWD/data:/data \
  nidavellir.jriyuu.io/jriyuu-labs/pokemon-agent:latest

Then:

• API: http://localhost:8765
• Dashboard ("Field Log"): http://localhost:8765/dashboard
• WebSocket event stream: ws://localhost:8765/ws

A ready-to-use docker-compose.yml for the standard layout lives in the workspace (projects/pokemon-agent/docker-compose.yml).

Serving a different ROM

Override the command — the server accepts any ROM the emulator supports (Game Boy .gb/.gbc via PyBoy; GBA via PyGBA):

docker run -d --name pokemon-agent \
  -p 8765:8765 \
  -v $PWD/roms:/roms:ro -v $PWD/data:/data \
  nidavellir.jriyuu.io/jriyuu-labs/pokemon-agent:latest \
  pokemon-agent serve --rom /roms/pokemon_firered.gba --data-dir /data --port 8765

Volumes and environment

Path	Purpose
/roms	ROM files (mounted read-only)
/data	Saves, game sessions, dashboard history (persists across restarts)

--load-state <name> on the serve command auto-loads a named save-state on startup.

What the agent gets

• GET /state — structured JSON: player (name, money, badges, position), party (species, levels, HP, moves, types), bag, battle, dialog, flags.
• GET /map/ascii — ground-truth walkability map derived from RAM (@ walkable, . ground, # off-limits) — navigate from real data, not pixels.
• GET /screenshot / /screenshot/grid — current frame (PNG) / frame with labelled A1..J9 grid + walkability tint.
• POST /action — execute actions, chained in one call:

curl -X POST localhost:8765/action \
  -H "Content-Type: application/json" \
  -d '{"actions": ["walk_up", "walk_up", "press_a"]}'

• POST /event — push narration (THINK/DECIDE/ACT/MILESTONE/ALERT) to the dashboard.
• GET/POST /objectives, /control — dashboard objectives and autopilot state.
• Game sessions — POST /games/new, GET /games, POST /games/<id>/load, GET /games/current; POST /save, POST /load, GET /saves for states.

Actions

Action	Description
press_a / press_b	Press A / B
press_start / press_select	Press Start / Select
walk_up / walk_down / walk_left / walk_right	Walk one tile
hold_a_30	Hold A for 30 frames
wait_60	Wait ~1 second
a_until_dialog_end	Press A until dialog closes

Full endpoint list

Endpoint	Method	Description
/	GET	Server info
/health	GET	Health check
/state	GET	Full game state JSON
/screenshot	GET	Current frame (PNG)
/screenshot/grid	GET	Frame + A1..J9 grid + walkability tint (PNG)
/screenshot/base64	GET	Current frame (base64 JSON)
/map/ascii	GET	ASCII walkability map
/minimap	GET	ASCII minimap
/action	POST	Execute game actions
/event	POST	Push agent narration to the dashboard
/objectives	GET/POST	Read/replace dashboard objectives
/control	GET/POST	Autopilot run state (running/paused/stopped)
/games	GET	List game sessions
/games/new	POST	Start a new game session
/games/{id}/load	POST	Load a game session
/games/{id}/hermes	POST	Bind a Hermes session id to a game
/games/{id}	DELETE	Delete a game session
/games/current	GET	Active session summary
/save	POST	Save emulator state
/load	POST	Load emulator state
/saves	GET	List saved states
/ws	WebSocket	Live event stream
/dashboard	GET	Web dashboard

Autopilot (driver)

The server is a passive API — it holds the emulator but does not play. To make the game play autonomously with an LLM, run the bundled driver in a second process:

pokemon-agent serve --rom /roms/pokemon_red.gb &   # terminal A
pokemon-agent play --port 8765                     # terminal B (needs the brain CLI on PATH)

START / PAUSE / STOP on the dashboard drive the loop; each turn is a full agent loop (grid screenshot + state → LLM → actions).

Rebuilding the image

Push to main (PR-first for changes after the initial import). Forgejo CI builds amd64 + arm64 and merges the manifests, tagging latest and <sha>. The Dockerfile is Dockerfile at the repo root.

To rebuild locally:

docker build -t nidavellir.jriyuu.io/jriyuu-labs/pokemon-agent:latest .

Upstream

Source: https://github.com/NousResearch/pokemon-agent — MIT licensed (see LICENSE). This fork adds the Docker packaging (Dockerfile, Forgejo CI, this README) for the jriyuu-labs registry; upstream changes are imported by re-cloning and re-committing.