MCP Tools

# Agora Conversational AI — MCP Recipe (Python)

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
[![Python](https://img.shields.io/badge/python-%3E%3D3.10-blue)](https://www.python.org/)
[![Bun](https://img.shields.io/badge/bun-latest-black)](https://bun.sh/)

The **mcp** recipe in the Agora Conversational AI recipes family. An OpenAI
Pipeline or Realtime model emits a tool call, Agora invokes the FastMCP server
mounted at `/mcp` in the same backend process, returns the result, and the model
speaks it.

Pipeline mode is **zero-key by default** because OpenAI, Deepgram STT, and
MiniMax TTS are Agora-managed. Set `OPENAI_API_KEY` to use your own OpenAI
credentials, and optionally override `OPENAI_BASE_URL` for a compatible
endpoint. Realtime mode uses the separate `OPENAI_REALTIME_API_KEY`. The mock
`get_time` tool needs no external credentials; replace it in
`server/src/mcp_server.py`.

**Distinct from `recipe-agent-tool-calling`**: in that recipe the tools run
inside the `llm/` endpoint. Here Agora orchestrates them via the MCP protocol —
the selected OpenAI path issues a tool call, Agora invokes `MCP_ENDPOINT`, and
the result flows back to the model.

## Prerequisites

- [Python 3.10+](https://www.python.org/)
- [Bun](https://bun.sh/)
- [Agora CLI](https://github.com/AgoraIO/cli) — makes generating an App ID + App Certificate easy
- [ngrok](https://ngrok.com/) — the backend (including the `/mcp` endpoint) must be publicly reachable so Agora cloud can call it

The same commands work on macOS, Linux, and Windows. On macOS/Linux, setup uses
`python3`; on Windows, it uses the Python launcher (`py`) or `python`. WSL and
virtualenv activation are not required.

## Run It

```bash
# 1. Install Python venv + web deps
bun run setup

# 2. Add Agora credentials to server/.env.local
agora login
agora project use <your-project>
agora project env write server/.env.local

# 3. Expose the backend publicly — Agora cloud calls /mcp on this tunnel
ngrok http 8000

# 4. Set MCP_ENDPOINT in server/.env.local (use whatever domain ngrok prints)
#    MCP_ENDPOINT=https://<your-tunnel>.ngrok-free.dev/mcp

# 5. Run the backend and the web frontend
bun run dev
```

Open [http://localhost:3000](http://localhost:3000) → **Start Conversation** →
ask "what time is it?".

### Working from a clone

If you cloned this repo (rather than scaffolding via the Agora CLI), the steps
above are complete as written: `bun run setup` creates the Python venv and
installs web dependencies, then `bun run dev` brings up both services. You
still need Agora credentials in `server/.env.local` and a public `MCP_ENDPOINT`
tunnel before a conversation can connect.

Services:

- Frontend — http://localhost:3000
- Backend + MCP server — http://localhost:8000 (including `/mcp`)
- API docs — http://localhost:8000/docs

## Deploy

Deploy `web` (Next.js) and `server` (a single publicly reachable FastAPI
process that also serves `/mcp`, so Agora cloud can reach `MCP_ENDPOINT`).
Set `AGENT_BACKEND_URL` in the web deployment so the Next rewrites reach the
backend.

A single-process Docker image is published to
`ghcr.io/AgoraIO-Conversational-AI/recipe-agent-mcp` on `v*` tags. It runs the
agent backend and the FastMCP server in one process on port 8000. Expose port
8000 publicly and point `MCP_ENDPOINT` at `<public-url>/mcp`.

**Co-public caveat**: because the `/mcp` endpoint is served on the same port as
the token endpoints, deploying this image publicly also exposes `/mcp`. For
production use, add authentication to the MCP server or deploy behind a gateway
that restricts `/mcp` access to Agora cloud IPs.

## Environment variables

Backend env file: [`server/.env.example`](https://github.com/AgoraIO-Conversational-AI/recipe-agent-mcp/blob/main/server/.env.example).

| Variable | Required | Default | Notes |
| --- | :---: | :---: | --- |
| `AGORA_APP_ID` | Yes | — | Agora Console → Project → App ID |
| `AGORA_APP_CERTIFICATE` | Yes | — | Agora Console → Project → App Certificate |
| `MCP_ENDPOINT` | Yes | — | **Public** URL of the `/mcp` endpoint (e.g. `https://<tunnel>/mcp`). Agora cloud calls it; cannot be `localhost`. |
| `OPENAI_MODEL` | | `gpt-4o-mini` | Pipeline model |
| `OPENAI_API_KEY` | | — | Optional Pipeline BYO API key; omit for Agora-managed mode |
| `OPENAI_BASE_URL` | | OpenAI chat completions URL | Optional OpenAI-compatible Pipeline endpoint override |
| `OPENAI_REALTIME_API_KEY` | Realtime only | — | OpenAI API key for Realtime mode |
| `OPENAI_REALTIME_MODEL` | | `gpt-realtime` | OpenAI Realtime model |
| `AGENT_GREETING` | | built-in | Optional opening line override |
| `PORT` | | `8000` | Agent backend port |
| `AGENT_BACKEND_URL` (web deploy) | Yes (deploy) | — | Required when deploying `web` |

## Commands

```bash
bun run setup            # install web deps + create server/ venv
bun run dev              # run backend (:8000, including /mcp) + web (:3000)

bun run doctor           # prerequisite check (no creds needed)
bun run doctor:local     # + .env.local + credentials + MCP_ENDPOINT checks

bun run verify           # web-only gate (no Agora creds needed)
bun run verify:local     # full local gate: backend compile + web build
bun run clean            # remove venv and build artifacts
```

Tests run standalone (no Agora cloud needed): `pytest` in `server/`, plus
`bun run verify` in `web/`. CI runs them on Linux/macOS/Windows × Python 3.10 &
3.13.

## Architecture

```
Browser (localhost:3000)
  │  fetch /api/*
  ▼
Next.js  ──rewrite──▶  Agent backend  (server/, localhost:8000)
                          │  starts selected OpenAI path + typed mcp_servers
                          │  also serves FastMCP at /mcp (same process)
                          ▼
                       Agora ConvoAI Cloud
                          │  Pipeline: Deepgram → OpenAI LLM → MiniMax
                          │  Realtime: OpenAI Realtime MLLM
                          │  selected model emits tool call
                          │  POST <MCP_ENDPOINT>   (streamable-http)
                          ▼
                       FastMCP server at /mcp  (same process, same port)
                          │  returns tool result → selected model speaks it
                          ▼
                       Agora ConvoAI Cloud → user hears speech
                                          → RTM transcript / metrics → web UI
```

The browser only ever calls Next `/api/*`, which rewrites to the agent backend.
The agent backend owns Agora tokens and agent lifecycle. The **FastMCP server**
is mounted in the same process on the same port — `ngrok http 8000` exposes
both. See [ARCHITECTURE.md](https://github.com/AgoraIO-Conversational-AI/recipe-agent-mcp/blob/main/ARCHITECTURE.md).

## What You Get

- A **Next.js** web client (:3000) that drives the RTC/RTM lifecycle and only
  ever calls `/api/*`.
- A **FastAPI** agent backend (:8000) that owns Agora token generation and the
  agent session lifecycle.
- The `/api/get_config` · `/api/startAgent` · `/api/stopAgent` contract between
  the web client and the backend (Next rewrites, no Route Handlers).
- Selectable managed OpenAI Pipeline and OpenAI Realtime MLLM paths with typed
  `mcp_servers` configuration and tool execution enabled.
- A **zero-key mock** MCP server mounted in-process; default Pipeline mode runs
  with no LLM API key and only one port to expose.

## How It Works

1. The browser calls `/api/get_config`, which Next rewrites to the backend; the
   backend mints an Agora token from `AGORA_APP_ID` + `AGORA_APP_CERTIFICATE`.
2. The browser joins the RTC channel, then calls `/api/startAgent` with
   `agentMode`; the backend attaches `mcp_servers` to managed OpenAI in Pipeline
   mode or OpenAI Realtime MLLM in Realtime mode.
3. Pipeline mode uses managed Deepgram STT and MiniMax TTS. Realtime mode handles
   audio directly in the MLLM.
4. When the model emits a tool call (e.g. `get_time`), Agora cloud issues a
   streamable-HTTP request to `MCP_ENDPOINT`. The FastMCP server (mounted at
   `/mcp` in the same process) runs the tool and returns the result.
5. Agora feeds the tool result back to the model. Pipeline mode uses MiniMax TTS
   for the reply; Realtime mode speaks it directly.
6. `/api/stopAgent` ends the session.

### Replacing the mock

Add tools in [`server/src/mcp_server.py`](https://github.com/AgoraIO-Conversational-AI/recipe-agent-mcp/blob/main/server/src/mcp_server.py). Each
function decorated with `@mcp.tool()` is automatically registered. The mock
`get_time` tool needs no external credentials — replace or extend it with your
own logic.

To verify tool arguments in either mode, ask "What time is it in 12-hour
format?" and then "Tell me in 24-hour format." The backend logs the selected
`time_format` and the returned server time.

## Repo Map

- `web/` — Next.js frontend (:3000); RTC/RTM lifecycle and UI.
- `server/` — FastAPI agent backend (:8000); Agora tokens + agent lifecycle,
  selectable OpenAI paths with `mcp_servers`, FastMCP server mounted at `/mcp`.
- `ARCHITECTURE.md` — system shape and component boundaries.
- `AGENTS.md` — guide for coding agents working in this repo.

## Troubleshooting

| Problem | Fix |
| --- | --- |
| Agent starts but never responds to "what time is it?" | `MCP_ENDPOINT` is not public or the `/mcp` path is wrong. Use your ngrok URL. |
| `doctor:local` warns about localhost | Replace the local URL with your public tunnel URL. |
| Local calls fail under a global proxy | Configure the proxy to send `127.0.0.1` and `localhost` DIRECT. |

## More Docs

- [ARCHITECTURE.md](https://github.com/AgoraIO-Conversational-AI/recipe-agent-mcp/blob/main/ARCHITECTURE.md)
- [AGENTS.md](https://github.com/AgoraIO-Conversational-AI/recipe-agent-mcp/blob/main/AGENTS.md)

## License

Released under the [MIT License](https://github.com/AgoraIO-Conversational-AI/recipe-agent-mcp/blob/main/LICENSE).