# Retrieval-Augmented Generation

> Ground the agent's answers in your own knowledge base.

Retrieve from an in-app corpus and inject context so the voice agent answers from your documents.

- Author: Agora
- Updated: 2026-06-12
- Difficulty: Intermediate
- Categories: voice-ai
- Platforms: Python, TypeScript
- Use cases: RAG, Knowledge Base, Voice AI
- Capabilities: Conversational AI, Voice Agent, RAG, Retrieval
- [Recipe page](https://recipes.agora.io/recipes/rag)
- [Implementation repository](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rag)
- [Source recipe](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rag/blob/main/README.md)
- [Raw source Markdown](https://raw.githubusercontent.com/AgoraIO-Conversational-AI/recipe-agent-rag/main/README.md)
- [All recipes](https://recipes.agora.io/llms.txt)

## Use with Agora CLI

agora init my-rag --recipe rag

## Implementation guide

# Agora Conversational AI — RAG Recipe (Python)

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
[![Python](https://img.shields.io/badge/python-%3E%3D3.10-blue)](https://www.python.org/)
[![Bun](https://img.shields.io/badge/bun-latest-black)](https://bun.sh/)

The **rag** recipe in the Agora Conversational AI recipes family. The agent's LLM
stage is pointed at the `/llm` endpoint that lives inside the same backend process:
on each user query the endpoint retrieves the best-matching document from a small
in-code corpus and grounds its reply in it ("Based on our docs: …"). STT (Deepgram
nova-3) and TTS (MiniMax) stay Agora-managed.

This repo ships a **zero-key mock** RAG endpoint so you can run the full
STT → RAG LLM → TTS pipeline immediately. Retrieval is real code; generation is
mocked. Swap `CORPUS` and `retrieve()` in `server/src/llm.py` for a real vector
store when you are ready.

## Prerequisites

- [Python 3.10+](https://www.python.org/)
- [Bun](https://bun.sh/)
- [Agora CLI](https://github.com/AgoraIO/cli) — makes generating an App ID + App Certificate easy
- [ngrok](https://ngrok.com/) — the backend (including `/llm`) must be publicly reachable so Agora cloud can call it

The same commands work on macOS, Linux, and Windows. On macOS/Linux, setup uses
`python3`; on Windows, it uses the Python launcher (`py`) or `python`. WSL and
virtualenv activation are not required.

## Run It

```bash
# 1. Install Python venv + web deps
bun run setup

# 2. Add Agora credentials (CLI), or edit server/.env.local by hand
agora login
agora project use <your-project>          # select which project to use (you may have several)
agora project env write server/.env.local # writes App ID/Certificate; keeps your CUSTOM_LLM_* lines

# 3. Expose the backend publicly (Agora cloud calls /llm/chat/completions directly)
ngrok http 8000

# 4. Add the tunnel URL to server/.env.local (use whatever domain ngrok prints —
#    today that is usually *.ngrok-free.dev)
#    CUSTOM_LLM_URL=https://<your-tunnel>.ngrok-free.dev/llm/chat/completions

# 5. Run the backend and web
bun run dev
```

Open [http://localhost:3000](http://localhost:3000) → **Start Conversation** → ask
about refunds, business hours, shipping, or warranty.

### Working from a clone

If you cloned this repo (rather than scaffolding via the Agora CLI), the steps
above are complete as written: `bun run setup` creates the Python venv and installs
web dependencies, then `bun run dev` brings up the backend and web. You still need
Agora credentials in `server/.env.local` and a public `CUSTOM_LLM_URL` tunnel
before a conversation can connect.

Services:

- Frontend — http://localhost:3000
- Backend (+ /llm) — http://localhost:8000
- API docs — http://localhost:8000/docs

## Deploy

Deploy `web` (Next.js) and `server` (a single publicly reachable FastAPI process
that also serves `/llm/chat/completions`). Set `AGENT_BACKEND_URL` in the web
deployment so the Next rewrites reach the backend.

A Docker image is published to `ghcr.io/AgoraIO-Conversational-AI/recipe-agent-rag`
on `v*` tags. It runs a single process on port 8000 — no second port needed. Point
`CUSTOM_LLM_URL` at `<public-url>/llm/chat/completions`. A local `docker run` still
needs a tunnel, because Agora cloud cannot reach `localhost`.

> **co-public caveat:** the backend serves both agent tokens and the `/llm` endpoint
> on the same public URL. This is intentional for the mock: in production, move RAG
> logic to a dedicated service and point `CUSTOM_LLM_URL` there.

## Environment variables

Backend env file: [`server/.env.example`](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rag/blob/main/server/.env.example).

| Variable | Required | Default | Notes |
| --- | :---: | :---: | --- |
| `AGORA_APP_ID` | ✅ | — | Agora Console → Project → App ID |
| `AGORA_APP_CERTIFICATE` | ✅ | — | Agora Console → Project → App Certificate |
| `CUSTOM_LLM_URL` | ✅ | — | **Public** URL of the `/llm/chat/completions` endpoint. Agora cloud calls it; cannot be `localhost`. |
| `CUSTOM_LLM_API_KEY` | ✅ | `any-key-here` | Forwarded by Agora cloud as `Authorization: Bearer`. Required by the `CustomLLM` vendor. |
| `CUSTOM_LLM_MODEL` | | `rag-mock` | Model name passed to your endpoint |
| `AGENT_GREETING` | | built-in | Optional opening line override |
| `RAG_TOP_K` | | `1` | Number of corpus docs to retrieve per query |
| `AGENT_BACKEND_URL` (web deploy) | ✅ | — | Required in a deployed `web` app when proxying to the backend |

## Commands

```bash
bun run setup            # install web deps + create server/ venv
bun run dev              # run backend (:8000, including /llm) + web (:3000)

bun run doctor           # prerequisite check (no creds needed)
bun run doctor:local     # + .env.local + credentials + CUSTOM_LLM_URL checks

bun run verify           # web-only gate (no Agora creds needed)
bun run verify:local     # full local gate: backend compile + smoke tests + web build
bun run clean            # remove venv and build artifacts
```

Tests run standalone (no Agora cloud needed): `pytest` in `server/`, plus
`bun run verify` in `web/`. CI runs them on Linux/macOS/Windows × Python 3.10 & 3.13.

## Architecture

```
Browser (localhost:3000)
  │  fetch /api/*
  ▼
Next.js  ──rewrite──▶  Agent backend  (server/, localhost:8000)
                          │  starts agent session (CustomLLM vendor)
                          │  also serves /llm/chat/completions (same process)
                          ▼
                       Agora ConvoAI Cloud
                          │  POST <CUSTOM_LLM_URL>   (Authorization: Bearer)
                          ▼
                       /llm endpoint  (server/src/llm.py, mounted at /llm)
                          ▲  public via ngrok tunnel (ngrok http 8000)
```

The browser only ever calls Next `/api/*`, which rewrites to the agent backend.
The agent backend owns Agora tokens and agent lifecycle. The **RAG LLM endpoint**
is mounted at `/llm` in the same process — expose port 8000 publicly and set
`CUSTOM_LLM_URL=<tunnel>/llm/chat/completions`. See [ARCHITECTURE.md](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rag/blob/main/ARCHITECTURE.md).

## What You Get

- A **Next.js** web client (:3000) that drives the RTC/RTM lifecycle and only
  ever calls `/api/*`.
- A **FastAPI** agent backend (:8000) that owns Agora token generation and the
  agent session lifecycle.
- The `/api/get_config` · `/api/startAgent` · `/api/stopAgent` contract between
  the web client and the backend (Next rewrites, no Route Handlers).
- The `/llm` endpoint retrieves the best-matching document from an in-code corpus
  and grounds the reply in it; Agora cloud receives only the final spoken response.
- A **zero-key mock** so the full pipeline runs with no LLM API key.

## How It Works

1. The browser calls `/api/get_config`, which Next rewrites to the backend; the
   backend mints an Agora token from `AGORA_APP_ID` + `AGORA_APP_CERTIFICATE`.
2. The browser joins the RTC channel, then calls `/api/startAgent`; the backend
   starts an agent session using the `CustomLLM` vendor pointed at
   `CUSTOM_LLM_URL`.
3. The user speaks. Agora runs STT (Deepgram nova-3), then sends the transcript
   to the `/llm` endpoint as an OpenAI `POST /chat/completions` request,
   forwarding `CUSTOM_LLM_API_KEY` as `Authorization: Bearer`.
4. Inside the endpoint, `retrieve()` scores the user query against the in-code
   `CORPUS` and returns the top-`RAG_TOP_K` documents. `run_agent_turn()` grounds
   the reply in those documents and streams the result in the OpenAI SSE chunk
   format ("Based on our docs: …").
5. Agora runs TTS (MiniMax) on the streamed reply and plays it back in the
   channel.
6. `/api/stopAgent` ends the session.

### Replacing the mock

Swap `CORPUS` and `retrieve()` in [`server/src/llm.py`](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rag/blob/main/server/src/llm.py) for a
real vector store (e.g. ChromaDB, pgvector, Pinecone). The `run_agent_turn()`
function and the OpenAI streaming contract must remain unchanged. A production
endpoint should also validate the `Authorization: Bearer` header.

## Repo Map

- `web/` — Next.js frontend (:3000); RTC/RTM lifecycle and UI.
- `server/` — FastAPI agent backend (:8000); Agora tokens + agent lifecycle, `CustomLLM` vendor, and `/llm` mock endpoint.
- `ARCHITECTURE.md` — system shape and component boundaries.
- `AGENTS.md` — guide for coding agents working in this repo.

## Troubleshooting

| Problem | Fix |
| --- | --- |
| Agent starts but never speaks | `CUSTOM_LLM_URL` is not public or omits `/llm/chat/completions`. Use your ngrok URL. |
| `doctor:local` warns about localhost | Replace the local URL with your public tunnel URL. |
| Local calls fail / hang under a global proxy (Clash, etc.) | Configure it to send `127.0.0.1`, `localhost`, and RFC-1918 ranges DIRECT. |

## More Docs

- [ARCHITECTURE.md](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rag/blob/main/ARCHITECTURE.md)
- [AGENTS.md](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rag/blob/main/AGENTS.md)

## License

Released under the [MIT License](https://github.com/AgoraIO-Conversational-AI/recipe-agent-rag/blob/main/LICENSE).


## Recipe prompt

You are implementing the "Retrieval-Augmented Generation" recipe in this project.

Read the recipe markdown first:
https://raw.githubusercontent.com/AgoraIO-Conversational-AI/recipe-agent-rag/main/README.md

Use the source repository for cross-reference:
https://github.com/AgoraIO-Conversational-AI/recipe-agent-rag

Build this recipe into the user's app using the markdown as the implementation guide. Inspect related source files through the repository links when the recipe points to them. Ask before installing new dependencies.