AIMC — AI agents in Minecraft (NeoForge 1.21.1)

A NeoForge mod (client + server) that lets an AI agent running on your machine play alongside you:

• Agent NPC — every player can spawn one agent entity. Everyone can see it, but only its owner can command it, open its inventory, hand it items or hurt it, and its chat goes to its owner only (server option publicChat makes chat public). It walks, mines, places blocks, fights hostile mobs, picks up drops, follows players, carries a 27-slot inventory and talks in chat. Ownership is enforced server-side, so it is multiplayer-safe.
• Local API + MCP server — the client opens http://127.0.0.1:25580 (loopback only). AI agents drive the NPC and read the world through plain JSON endpoints or through the Model Context Protocol endpoint at /mcp, which works directly with Claude Code and other MCP clients.
• In-game tmux terminal — press K to list your tmux sessions, create one (default command: claude) and use it full-screen inside Minecraft. Keys are forwarded to tmux, so interactive TUIs work.
• Per-client configuration — port, token, tmux binary/defaults, terminal size and event log are per client (config/aimc-client.toml, editable in-game under Mods → AIMC → Config). Server rules live in config/aimc-common.toml on the server.

The mod is required on both sides (server and every client).

Build

Requires JDK 21. CI (Forgejo Actions, .forgejo/workflows/build.yml) builds every push and pull request, keeps the jar as a downloadable artifact for 30 days, and attaches it to a release for tags named v* (git tag v0.1.0 && git push --tags).

./gradlew build            # jar in build/libs/
./gradlew runClient        # dev client
./gradlew runServer        # dev server (accept the EULA in run/ first)

Quick start

1. Install the jar on the server and on your client, join the server.
2. /agent spawn Bob — the agent appears next to you.
3. Press K, click Create — a tmux session named aimc running claude opens in-game.
4. In a terminal (or from the game with /aimc mcp, which prints a copyable command), register the MCP server:

   claude mcp add --transport http minecraft http://127.0.0.1:25580/mcp
   

5. Tell Claude something like "Spawn my agent, then mine the oak log at 12 64 -8 and bring me the wood." Players can talk to the AI by chatting: the AI sees chat via get_events / wait_for_events, and the agent can answer with agent_say.

Talking to the agent

The mod has no built-in AI; the agent's "voice" is whatever you connect over MCP. With Claude Code:

1. In-game: /agent spawn Bob, then /aimc mcp and copy the claude mcp add line (or run it in a terminal).
2. Press K → Create to start claude in a tmux session inside the game, and run /minecraft:play there. That MCP prompt puts Claude into a loop: wait_for_events → act on chat addressed to it → answer with agent_say → report task results. Chat events carry mention: true when a message starts with the agent's name or contains @Bob, so it knows what is meant for it.
3. Talk in chat: @Bob follow me, Bob, mine me 10 iron, @Bob what's in the chest at 100 64 -20?
4. /agent narrate on makes the agent announce, as itself, what it starts and how it went (" Going mining for diamonds (5).", " Done: mined 5 diamond_ore."). Owner-only status still shows in grey.

Any other MCP-capable agent works the same way; the REST API (/events/wait, POST /agent) is enough for a hand-written loop.

In-game commands

Command	Effect
/agent spawn [name] / /agent despawn	Create / remove your agent (despawn returns its items to you)
/agent status	Position, health, task, queue, inventory count
/agent help	List all subcommands
/agent goto <x y z> · /agent mine <x y z> · /agent place <x y z> <item>	Tasks
/agent gather <block> [count] [radius]	Autonomously mine blocks of a type (diamond_ore, oak_log, #minecraft:logs); ores are prospected by digging down and strip-mining
/agent follow [player] · /agent attack [radius] · /agent collect [radius]	Tasks
/agent containers [r] · /agent inspect <pos> · /agent deposit <pos> [item] [n] · /agent withdraw <pos> [item] [n] · /agent smelt <pos> <item> [n] · `/agent home	clear`
`/agent narrate [on	off]`
/agent stop · /agent come (walks to you) · /agent say <msg> · /agent craft <item> [count] · /agent equip <item> · /agent give [item] · /agent rename <name>	Immediate actions
/agent json {"type":...}	Raw command (same format as the API)
/aimc terminal [session] · /aimc sessions	Open the tmux UI / list sessions (client)
`/aimc api [restart	stop]·/aimc mcp·/aimc events [clear]`

Right-click the agent with an empty hand to open its inventory like a chest (owner only, within 8 blocks). Right-click holding an item to hand it over; sneak + right-click with an empty hand to take everything back.

Agent command format

All commands are JSON objects with a type. Task commands accept "queue": true to append after the current task instead of replacing it. Failed tasks clear the queue.

type	fields	notes
spawn	name	one agent per player
despawn, status, stop, come
goto	x y z, arrive	walks; fails with no path / stuck
follow	player	default: owner; runs until stopped
mine	x y z	auto-equips the best tool; drops go to the agent inventory
gather	block, count, radius, y, explore	block id, #tag, or a friendly name (diamonds → all diamond ores, wood → all logs, spruce → spruce logs, ore → any ore); mines the nearest exposed matches within radius (no x-ray). For ores (or with y) it prospects: digs a staircase to the typical depth (diamonds −58, iron 16, gold −16, coal 96, ...), strip-mines 1×2 tunnels in 48-block legs, scans the freshly exposed walls after every step, follows veins, seals lava/water, bridges holes with cobblestone and places torches every 8 blocks (crafting them from coal + sticks when it can). It needs a pickaxe and will craft one from what it carries (cobblestone/planks + sticks, or logs) before digging. count default 16, radius 24.
place	x y z, item	item must be in the agent inventory
attack	radius, target (uuid)	hostile mobs only, never players
collect	radius	picks up item drops
say	message	<AgentName> message in chat, to the owner (or everyone with publicChat)
look	x y z
containers	radius	visible containers near the agent: chests, barrels, furnaces, hoppers, shulkers and any modded block exposing an item handler (AE2 interfaces/buses, drawers, ...)
inspect	x y z	walk to a container and report its contents (in the task_done event / result with wait)
deposit	x y z, item, count, keepEssentials	agent inventory → container
withdraw	x y z, item, count	container → agent inventory
smelt	x y z, item, count	load a furnace with the item and fuel from its inventory, wait for it, collect the output
home	x y z / clear	container where gather dumps loot whenever its inventory is full (keeps tools, torches, cobblestone)
craft	item, count	real recipes from its inventory; sub-ingredients are crafted recursively; ingredients are also pulled from containers within 4 blocks (chests, AE2 interfaces, ...); 3x3 recipes need a crafting table within 4 blocks (it places one from its inventory, crafting it from planks if needed). Smelting goes through the smelt task.
equip	item, hand
give	item, count	owner must be within 6 blocks
rename	name
narrate	on	announce task starts/results in public chat as the agent

Every command is answered with {ok, message, data?}. Tasks finish asynchronously and produce a task_done / task_failed event (with the cid of the request if you set one). Add "wait": true (and optionally waitTimeoutMs) to any task command over the API to block until it finishes and get {completed, ok, message, result}; the MCP container tools do this by default.

HTTP API (client-local)

Base URL http://127.0.0.1:25580 (configurable). If a token is configured, send Authorization: Bearer <token>.

Route	Description
GET /	Route list
GET /state	Player, world time, latest agent status
GET /events?since=&limit=&clear=	Event log: chat, event (task_done, task_failed, damaged, died, spawned, despawned), error
GET /events/wait?since=&timeout=	Long-poll until a new event arrives
DELETE /events	Clear the log
GET /agent	Fresh agent status from the server
POST /agent	Body = command object (see above)
POST /chat	{"message": "..."} — chat, or a /command
GET /blocks?x&y&z&r&air	Blocks in a cube (r ≤ 8) around a point (default: player), loaded chunks only
GET /entities?r	Entities near the player
GET /players	Online players
GET /tmux/sessions · POST /tmux/sessions {name,command} · DELETE /tmux/sessions/{name}	tmux sessions
POST /tmux/send {session,text,enter,keys} · GET /tmux/capture?session=	Type into / read a session
POST /mcp	MCP Streamable HTTP endpoint (JSON responses; no SSE)

MCP tools mirror the above: get_state, get_events, wait_for_events, agent_spawn, agent_goto, agent_mine, agent_gather, agent_place, agent_attack, agent_collect, agent_follow, agent_say, agent_stop, agent_come, agent_craft, agent_containers, agent_inspect, agent_deposit, agent_withdraw, agent_smelt, agent_home, agent_equip, agent_give, agent_look, agent_rename, agent_command, send_chat, get_blocks, get_entities, get_players, tmux_*.

Terminal keys

• Everything, including Esc, Ctrl+…, Alt+…, arrows and function keys, goes to tmux.
• F10 closes the terminal. Cmd+V / Ctrl+Shift+V pastes. Mouse wheel scrolls tmux history.
• The tmux window is resized to fit the screen (window-size manual) and released when the screen closes.

Server config (aimc-common.toml)

allowAgents, allowMining, allowPlacing, allowAttacking, allowTeleport (default off), publicChat (default off), invulnerable, reach, moveSpeed, mineSpeedMultiplier, statusIntervalTicks, maxQueuedTasks.

Notes and limitations

• Agents are regular entities: they only act while their chunk is loaded. If you spawn a new agent while the old one sits in an unloaded chunk, the old one is retired (items dropped) when its chunk loads again.
• No cheats by default: the agent walks everywhere, only exposed blocks are found by gather/get_blocks (no x-ray), and items only change hands within 6 blocks. allowTeleport in the server config re-enables teleporting.
• get_blocks reads the client's loaded chunks around the player, not around the agent.
• Block placement uses the block's default state (no orientation for stairs, logs, etc.).
• The API binds to loopback by default. Do not expose it to a network without a token.