Delegated Reasoning

# Delegated reasoning: a voice agent that stays responsive while a second model thinks

A full-duplex **GPT Live** voice agent that hands the hard thinking and tool use to a
separate **Responses model** (`gpt-6-astra`) over the Agora Conversational AI Engine. The
delegate drives a slow tool over **MCP**, so the voice loop keeps talking and listening while
the reasoning happens off the critical path.

Built on the official [`agora-agents`](https://www.npmjs.com/package/agora-agents) SDK and
Next.js, adapted from the
[OpenAI GPT Live + Next.js sample](https://github.com/AgoraIO-Community/OpenAI-Agora-Voice-Agents-NextJS).

## The idea

A single real-time model that also reasons and calls tools goes silent on the turn it stops to
think. Splitting the work fixes that:

- **GPT Live (front model)** owns the live audio: it listens, talks, backchannels, and decides
  when a turn needs real work. Set with `.withMllm(new OpenAIGPTLive(...))`.
- **The Responses model (`gpt-6-astra`)** is a reasoning delegate the front model consults
  mid-conversation. It picks candidate cities, calls the tools, compares results, and composes
  the answer, which comes back out through GPT Live's voice. Enabled with
  `delegation: 'responses'` + `responsesModel`.
- **Tools over MCP**: a small trip-planner MCP server exposes `get_forecast` (real Open-Meteo
  lookups, a few seconds) and `save_trip` (writes the chosen trip to disk). Attached with
  `mcpServers` + `.withTools(true)`.

## The scenario

> "I want to go somewhere sunny this weekend."

The front model backchannels and says it's checking; the delegate looks up several cities in
one `get_forecast` call, compares them, and recommends one; you pick a city and it calls
`save_trip`; change your mind and it reuses the same trip id so there's one record, not two.

## Prerequisites

- An Agora project with the **GPT Live (live-models) preview** feature enabled — App ID + App
  Certificate.
- An **OpenAI key** with access to GPT Live and the delegate model (`gpt-6-astra`).
- Node 20+ and `pnpm`. Python 3.10+ to run the MCP server.

## Run it

### 1. The MCP server (tools)

```bash
cd mcp-server
pip install -r requirements.txt
python server.py            # binds 0.0.0.0:8787, MCP endpoint at /mcp
```

The agent runs in **Agora's cloud**, so it must reach this server over the public internet.
Expose it at a public HTTPS URL (a tunnel such as cloudflared/ngrok, or a real deployment) and
use that URL as `NEXT_MCP_ENDPOINT` below. `save_trip` writes to `mcp-server/output/` by
default (`OUTPUT_DIR` to change).

### 2. The app

```bash
cp env.local.example .env.local   # fill in App ID, certificate, OpenAI key, NEXT_MCP_ENDPOINT
pnpm install
pnpm run dev                       # http://localhost:3000
```

Open the app, allow the microphone, and talk. Leave `NEXT_MCP_ENDPOINT` empty to demo
delegation without tools.

## How the delegation is wired

In `app/api/invite-agent/route.ts`:

```ts
const agent = new Agent({
  client,
  advancedFeatures: { enable_rtm: true, enable_tools: Boolean(MCP_ENDPOINT) },
  parameters: { audio_scenario: 'chorus', data_channel: 'rtm', enable_error_message: true, enable_metrics: true },
}).withMllm(
  new OpenAIGPTLive({
    apiKey: OPENAI_API_KEY,
    model: 'gpt-live-1',
    prompt: FRONT_INSTRUCTIONS,
    greeting: GREETING,
    delegation: 'responses',           // delegate reasoning + tools
    responsesModel: 'gpt-6-astra',      // the delegate model
    outputBufferMs: -1,                 // lowest first-audio latency
    ...(MCP_ENDPOINT && {
      mcpServers: [{ name: 'trip-planner', endpoint: MCP_ENDPOINT, transport: 'streamable_http', timeout_ms: 30000 }],
    }),
    params: { responses_params: { instructions: BACKEND_INSTRUCTIONS } },
  }),
);
const readyAgent = MCP_ENDPOINT ? agent.withTools(true) : agent;   // expose MCP tools to the delegate
```

The front and backend models are prompted separately: the front prompt is the conversational
persona and when to hand off; the backend prompt (sent via `responses_params.instructions`) is
the trip-planning logic.

## What "working" means

- **It does not guess while waiting.** No temperature, forecast, or recommendation is spoken
  before the tool returns; the front model fills the wait ("let me hand this to gpt-6").
- **It stays interactive during the wait.** Speak again while the lookup runs and it responds,
  because the reasoning is off the voice loop.
- **A correction leaves one record.** After "actually, Seville" `save_trip` reuses the trip id,
  so there is one file, not two.
- **It does not re-delegate to repeat itself.** "What did you say again?" produces no new tool
  calls.

## Notes and limitations

- **GPT Live is a preview provider**, routed to Agora's preview gateway by the SDK. Your App ID
  must have the `live-models` preview feature.
- **No avatar with GPT Live.** The `agora-agents` SDK does not allow an avatar with an MLLM
  (`withMllm()` + `withAvatar()` throws); avatars require the cascading ASR + LLM + TTS
  pipeline. This recipe is voice-only.
- `save_trip` is a stand-in for any real action — booking, ordering, filing a ticket. Swap that
  one tool and the rest is unchanged.