# Voice RPG

> Play a voice-driven tabletop RPG with an AI dungeon master.

A managed-LLM dungeon master backed by a stateful MCP game server with character, combat, and dice tools.

- Author: Agora
- Updated: 2026-06-12
- Difficulty: Advanced
- Categories: voice-ai
- Platforms: Python, TypeScript
- Use cases: Gaming, Entertainment, Voice AI
- Capabilities: Conversational AI, Voice Agent, MCP, Game State
- [Recipe page](https://recipes.agora.io/recipes/voice-rpg)
- [Implementation repository](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rpg)
- [Source recipe](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rpg/blob/main/README.md)
- [Raw source Markdown](https://raw.githubusercontent.com/AgoraIO-Conversational-AI/recipe-agent-rpg/main/README.md)
- [All recipes](https://recipes.agora.io/llms.txt)

## Use with Agora CLI

agora init my-voice-rpg --recipe voice-rpg

## Implementation guide

# Agora Conversational AI — RPG Gaming Recipe (Python)

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
[![Python](https://img.shields.io/badge/python-%3E%3D3.10-blue)](https://www.python.org/)
[![Bun](https://img.shields.io/badge/bun-latest-black)](https://bun.sh/)

The **RPG gaming** recipe in the Agora Conversational AI recipes family. A voice
RPG where a managed-LLM **Dungeon Master** narrates the adventure and calls
**game tools** mounted in the same backend process. The player speaks; the DM
resolves every mechanic — dice rolls, combat, loot, inventory — through 6
self-contained MCP tools backed by SQLite. STT (Deepgram) and TTS (MiniMax) are
Agora-managed.

This recipe is **zero-key**: OpenAI is Agora-managed (no `OPENAI_API_KEY`
needed, though you may supply your own). The FastMCP game server is mounted
in-process — one backend, one port (**:8000**). The full pipeline runs locally
with only Agora credentials and a public tunnel.

**Distinct from `recipe-agent-tool-calling`**: in that recipe tools run inside
the `llm/` endpoint. Here Agora cloud orchestrates them on a **FastMCP server**
mounted inside the API server — Agora cloud calls it directly at `MCP_ENDPOINT`
(`<public-url>/mcp`).

## Prerequisites

- [Python 3.10+](https://www.python.org/)
- [Bun](https://bun.sh/)
- [Agora CLI](https://github.com/AgoraIO/cli) — generates App ID + Certificate
- [ngrok](https://ngrok.com/) — the `/mcp` path must be publicly reachable so
  Agora cloud can call it

The same commands work on macOS, Linux, and Windows. On macOS/Linux, setup uses
`python3`; on Windows, it uses the Python launcher (`py`) or `python`. WSL and
virtualenv activation are not required.

## Run It

```bash
# 1. Install Python venv + web deps
bun run setup

# 2. Add Agora credentials to server/.env.local
agora login
agora project use <your-project>
agora project env write server/.env.local

# 3. Expose the backend publicly — /mcp is served by the same process
ngrok http 8000

# 4. Set MCP_ENDPOINT in server/.env.local (use whatever domain ngrok prints)
#    MCP_ENDPOINT=https://<your-tunnel>.ngrok-free.dev/mcp

# 5. Run backend + frontend
bun run dev
```

Open [http://localhost:3000](http://localhost:3000) → **Start Conversation** →
say "I want to be a warrior" to create your hero, then explore and fight.

### Working from a clone

If you cloned this repo (rather than scaffolding via the Agora CLI), the steps
above are complete as written: `bun run setup` creates the Python venv and
installs web dependencies, then `bun run dev` brings up the backend and
frontend. You still need Agora credentials in `server/.env.local` and a public
`MCP_ENDPOINT` tunnel before a conversation can connect.

Services:

- Frontend — http://localhost:3000
- Backend + MCP game server — http://localhost:8000
- API docs — http://localhost:8000/docs
- MCP endpoint — http://localhost:8000/mcp

## Deploy

Deploy `web` (Next.js) and `server` (a single publicly reachable FastAPI
backend). Set `AGENT_BACKEND_URL` in the web deployment so the Next rewrites
reach the backend.

The backend must be publicly reachable so Agora cloud can call `/mcp`. A single
Docker image is published to
`ghcr.io/AgoraIO-Conversational-AI/recipe-agent-rpg` on `v*` tags. It runs one
process on port 8000 with the FastMCP game server mounted at `/mcp`.

## Environment variables

Backend env file: [`server/.env.example`](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rpg/blob/main/server/.env.example).

| Variable | Required | Default | Notes |
| --- | :---: | :---: | --- |
| `AGORA_APP_ID` | Yes | — | Agora Console → Project → App ID |
| `AGORA_APP_CERTIFICATE` | Yes | — | Agora Console → Project → App Certificate |
| `MCP_ENDPOINT` | Yes | — | **Public** URL ending in `/mcp` (e.g. `https://<tunnel>/mcp`). Agora cloud calls it; cannot be `localhost`. |
| `OPENAI_MODEL` | | `gpt-4o-mini` | Model name for the managed Dungeon Master LLM |
| `RPG_DB_PATH` | | `rpg.db` | Path to the SQLite database, relative to `server/` or absolute. Docker uses `/tmp/rpg.db`. |
| `RPG_SEED` | | — | Optional integer seed for deterministic dice (useful for testing) |
| `OPENAI_API_KEY` | | — | Optional — Agora manages the OpenAI key (keyless by default) |
| `AGENT_GREETING` | | built-in | Optional override for the DM's opening line |
| `PORT` | | `8000` | Agent backend port |
| `AGENT_BACKEND_URL` (web deploy) | Yes (deploy) | — | Required when deploying `web` |

## Commands

```bash
bun run setup            # install web deps + create server/ venv
bun run dev              # run backend (:8000) + web (:3000)

bun run doctor           # prerequisite check (no creds needed)
bun run doctor:local     # + .env.local + credentials + MCP_ENDPOINT checks

bun run verify           # web-only gate (no Agora creds needed)
bun run verify:local     # full local gate: backend compile + smoke tests + web build
bun run clean            # remove venvs and build artifacts
```

Tests run standalone: `pytest` in `server/`, plus `bun run verify` in `web/`.
CI runs them on Linux/macOS/Windows × Python 3.10 & 3.13.

## Architecture

```
Browser (localhost:3000)
  │  fetch /api/*
  ▼
Next.js  ──rewrite──▶  Agent backend  (server/, localhost:8000)
                          │  starts agent session (Dungeon Master LLM + mcp_servers)
                          │  also serves /mcp  (FastMCP game server, in-process)
                          ▼
                       Agora ConvoAI Cloud
                          │  user speech → Deepgram STT (managed)
                          │  Dungeon Master LLM (managed OpenAI, keyless) → emits tool call
                          │  POST <MCP_ENDPOINT>   (streamable-http)
                          ▼
                       FastMCP game server  (mounted at /mcp, same process)
                          │  public via ngrok tunnel on :8000
                          │  resolves dice/combat/inventory → returns result
                          ▼
                       Agora ConvoAI Cloud → DM narrates outcome
                                          → MiniMax TTS (managed) → user hears speech
                                          → RTM transcript / metrics → web UI
```

The browser only ever calls Next `/api/*`, which rewrites to the agent backend.
The agent backend owns Agora tokens, agent lifecycle, **and** the FastMCP game
server — all in one process on port 8000. See [ARCHITECTURE.md](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rpg/blob/main/ARCHITECTURE.md).

## What You Get

- A **voice RPG** where a managed-LLM Dungeon Master narrates the adventure and
  calls game tools — no UI to click, no state to manage client-side.
- A managed-LLM DM that narrates and calls **6 self-contained MCP tools**: dice
  rolling, character creation, combat rounds, spells, fleeing, and inventory reads.
- **SQLite** backs dice, combat, and inventory — no external game server or
  database required.
- **Zero-key**: OpenAI is Agora-managed and the game engine needs no external
  credentials. The full pipeline runs locally with only Agora credentials and a
  public tunnel.

| Tool | When the DM calls it |
| --- | --- |
| `create_character(char_class)` | Player picks or changes their class (warrior/mage/rogue/cleric) |
| `get_character()` | Player asks about their stats, HP, gold, or inventory |
| `start_encounter()` | Player looks for a fight or the story leads into danger |
| `attack()` | Player attacks the current enemy |
| `cast_spell(name)` | Player casts their class spell |
| `flee()` | Player runs from combat |

Each tool opens its own SQLite connection, resolves the full action (including
dice rolls and counterattacks), and returns a plain-English result for the DM to
narrate. No chaining — one player utterance maps to at most one tool call.

## How It Works

1. The browser calls `/api/get_config`; the backend mints an Agora token.
2. The browser joins the RTC channel, then calls `/api/startAgent`; the backend
   starts an agent session using the managed `OpenAI` vendor with `mcp_servers`
   pointing at the public `MCP_ENDPOINT` (`<tunnel>/mcp`) and `enable_tools: true`.
3. The user speaks (e.g. "I want to be a warrior"). Agora runs STT (Deepgram)
   and sends the transcript to the managed Dungeon Master LLM.
4. The DM decides to call `create_character("warrior")`. Agora cloud issues a
   streamable-HTTP request to `MCP_ENDPOINT`. The FastMCP server (mounted at
   `/mcp` in the same process) runs the tool and returns a narrative result string.
5. Agora feeds the tool result back to the DM LLM, which narrates it (e.g.
   "You are a warrior with 30 HP…"). Agora runs TTS (MiniMax) and plays it back.
6. Later tools (`start_encounter`, `attack`, `cast_spell`, `flee`) resolve combat
   in the same way — each tool is **self-contained** (dice rolled inside `game.py`,
   no tool-call chaining).
7. `/api/stopAgent` ends the session.

## Repo Map

- `web/` — Next.js frontend (:3000); RTC/RTM lifecycle and UI.
- `server/` — FastAPI agent backend (:8000); Agora tokens, Dungeon Master agent
  lifecycle, and the FastMCP game server (mounted at `/mcp`).
- `server/src/game.py` — pure game engine (SQLite, no MCP dependency, fully unit-testable).
- `server/src/mcp_server.py` — FastMCP wrapper exposing 6 game tools.
- `ARCHITECTURE.md` — system shape and component boundaries.
- `AGENTS.md` — guide for coding agents working in this repo.

## Troubleshooting

| Problem | Fix |
| --- | --- |
| DM greets but never calls a tool | `MCP_ENDPOINT` is not public or the `/mcp` path is wrong. Use your ngrok URL. |
| `doctor:local` warns about localhost | Replace the local URL with your public tunnel URL. |
| Local calls fail under a global proxy | Configure the proxy to send `127.0.0.1` and `localhost` DIRECT. |
| Tests fail with wrong dice outcomes | Set `RPG_SEED` to a fixed integer; the tests already do this automatically. |

## More Docs

- [ARCHITECTURE.md](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rpg/blob/main/ARCHITECTURE.md)
- [AGENTS.md](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rpg/blob/main/AGENTS.md)

## License

Released under the [MIT License](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rpg/blob/main/LICENSE).


## Recipe prompt

You are implementing the "Voice RPG" recipe in this project.

Read the recipe markdown first:
https://raw.githubusercontent.com/AgoraIO-Conversational-AI/recipe-agent-rpg/main/README.md

Use the source repository for cross-reference:
https://github.com/AgoraIO-Conversational-AI/recipe-agent-rpg

Build this recipe into the user's app using the markdown as the implementation guide. Inspect related source files through the repository links when the recipe points to them. Ask before installing new dependencies.