Kaun Katega Taarpati — Multiplayer Voice Game Show Recipe (Next.js)

A three-player Hinglish game show driven by one Agora Conversational AI agent. Three people scan a QR code, join on their phones, and become contestants. The host — an original character on a Sarvam voice — asks them paheliyan. Each correct answer cuts one of five wires. Six minutes on the clock. Stuck? Phone a Friend places a real PSTN call, and a voice on the other end reads the hint on loop until the lifeline expires.

What makes this recipe different

Most voice-agent recipes are one user, one agent, and the model is trusted to remember what happened. None of those hold here:

Distinct from recipe-agent-custom-llm: that recipe shows you how to point Agora at your own OpenAI-compatible endpoint. This one shows you what to do with the seat once you have it — inject ground truth, execute tools against it, and refuse whatever the model got wrong.

Distinct from recipe-agent-rpg: there, Agora cloud orchestrates tools on an MCP server. Here the tool loop runs inside the proxy, one hop earlier, so the same request that runs a tool also decides what the model is allowed to believe.

Prerequisites

• Node.js 20+
• An Agora project with an App Certificate, plus

RESTful API credentials (Console → Developer Toolkit → RESTful API)

• A Sarvam API key — one key covers both ASR

(Saaras) and TTS (Bulbul)

• An LLM provider key. Groq is the default.

Agora does not host an LLM — this is mandatory, not optional.

• cloudflared

— Agora's cloud must reach /api/llm, so localhost is not enough

• (Optional) A Vobiz account and a number you

own, for the Phone a Friend lifeline

Run It

# 1. Install
npm install

# 2. Credentials
cp .env.example .env.local     # then fill it in

# 3. Start the server and renew the tunnel in one step
npm run dev

npm run dev starts Next and renews the cloudflared quick tunnel, writes the new hostname into PUBLIC_BASE_URL, and verifies the server answers through it. These are deliberately one command: a quick tunnel's hostname changes on every start, and a stale PUBLIC_BASE_URL breaks the LLM proxy and every telephony webhook without erroring anywhere.

Open <http://localhost:3000> — the landing page runs the pre-flight check and opens a room. Then:

• Projector / laptop → /stage/<code> — the 3D set, the wires, the chyron
• Each phone → /join/<code> — scan the QR, enter a name, become a contestant
• Operator → /host/<code> — the host console and live programme feed

Before every run

npm run check          # typecheck + the engine rule assertions
npm run render:audio   # re-render hint clips and outcome stingers via Sarvam

and open /api/health, which reports what is missing. Both exist for one reason: this architecture has an unusual number of failures that are silent. A hint MP3 that is not on disk makes the telephony provider skip the audio, so Phone a Friend becomes forty-five seconds of dead air on a live call, with no error in any log.

Environment variables

Full annotated list: `.env.example`.

Architecture

  Phones (QR + name)          Projector / laptop            This server
  ──────────────────          ──────────────────            ───────────
  mic publish            ◄── SSE + POST ──►  Three.js set   Next.js
  Peer Talk toggle                           Agora RTC recv  ├─ game state (authoritative)
  level telemetry                            room speakers   ├─ derived clock
  wire pips                                  chyron          ├─ attribution
  Phone a Friend                                             ├─ /api/llm  (custom LLM endpoint)
        │                                                    ├─ tool handlers
        │                                                    ├─ 5 riddles + hints
        └────────── inbound PSTN call ◄──────────────────────┴─ telephony client

              Agora Conversational AI Engine
              ASR: Sarvam Saaras → LLM: /api/llm → TTS: Sarvam Bulbul

Agora's cloud calls /api/llm on every user turn. That endpoint — not the model — is where the game actually lives.

The three patterns worth stealing

1. The custom LLM endpoint is the spine

Agora streams whatever the LLM emits straight into TTS. Point llm.url at a provider directly and a tool call reaching Agora is read aloud as gibberish. Sitting in the middle means the proxy can:

1. call the model itself, non-streaming, so tool calls are trivial to read
2. execute tool calls against real server state
3. feed results back and loop, up to MAX_TOOL_ROUNDS (5)
4. stream only the final prose onward as OpenAI-shaped SSE chunks

// lib/agora-rest.ts — the field the whole build rests on
llm: {
  url: `${publicBaseUrl()}/api/llm?room=${encodeURIComponent(channel)}`,
  api_key: optional("PROXY_SHARED_SECRET", "unused"),  // Agora just needs non-empty
  system_messages: [{ role: "system", content: systemPrompt }],
  max_history: 40,
  greeting_message: greeting,
  params: { model: optional("LLM_MODEL", "..."), stream: true },
}

The cost is real: a tool round adds latency before the agent speaks. Agora's filler_words covers it — and in a game show under a countdown, that latency reads as suspense rather than lag, which is a free pass on the thing that usually damages voice demos.

2. The server owns the clock, and it is derived, never ticked

There is no setInterval decrementing a counter anywhere. secondsLeft() computes from startedAt, accumulated pause time, and the sum of penalties. Every reader — phones, projector, model, host console — derives the same number from the same facts, and hot reload cannot desync it.

The model is told the state every turn and never computes it. A language model asked to count seconds will invent a number, confidently, in front of an audience. So every request to /api/llm gets a freshly-built LIVE STATE system message: clock, wire statuses, who is live, what was just guessed, and the answer key for the active wire. It is regenerated per request rather than cached, because a stale clock is worse than no clock.

3. One live mic collapses the hardest problem in multiplayer voice

Every phone has a Peer Talk toggle, on by default. While it is on, that contestant's mic is not published to the channel — the host genuinely cannot hear them, and they argue about the answer through the air, like people in a room do. To speak to the host, you switch Peer Talk off.

When exactly one contestant is live, whatever the ASR heard is that person. Attribution becomes arithmetic rather than inference. It also means one open mic instead of three, so most of the room-speaker echo problem stops existing. Level-telemetry attribution survives as the fallback for when two people go live at once — which is now a deliberate act, and makes the arbitration beat clearer rather than muddier.

What You Get

• A three-publisher voice agent: one agent, remote_rtc_uids: ["*"], three

phones, and a wildcard that also covers late joiners and reconnects.

• Semantic answer checking, never string matching — the model judges whether

what was said means the answer, and the server refuses any cut that is not for the active wire.

• Eight tools the host acts through, all backed by authoritative state:

• A human escalation path off the channel: the lifeline dials a real phone,

and the friend hears a pre-rendered hint clip on loop, with explicit pauses between plays, until the lifeline expires.

• Barge-in tuned for a room, not a headset — VAD start-of-speech, semantic

end-of-speech, and a 3.5s silence timeout that follows broadcast pacing.

• RTM transcripts and metrics, which need both

advanced_features.enable_rtm: true and parameters.data_channel: "rtm". One without the other fails silently.

How It Works

1. The projector opens a room. Phones scan the QR and POST /api/room/<code>/join.
2. GET /api/agora/token mints an RTC token per phone; each publishes its mic.
3. POST /api/room/<code>/agent mints a second token for the agent uid and calls

Agora's /join, handing over the whole ASR/LLM/TTS config — including llm.url pointed back at this server.

1. A contestant switches Peer Talk off and speaks. Agora runs Sarvam Saaras with

language: "unknown", so "nariyal" or "coconut" or half of each all transcribe.

1. Agora POSTs the turn to /api/llm. The proxy prepends a fresh LIVE STATE

block and calls the upstream model with the eight tool definitions.

1. The model emits e.g. cut_wire("red", "p2"). The proxy executes it against

the store, which validates it, applies the rule, advances the clock, and emits an SSE event that every client renders.

1. The tool result goes back to the model, which narrates the outcome. Only that

prose is streamed on to Agora, which runs Sarvam Bulbul and plays it into the room.

1. For lines that must not be improvised — the lifeline announcement, the closing

line after the outcome stinger — the server calls POST /agents/<id>/speak directly with interruptable: false, skipping the model entirely.

Repo Map

Troubleshooting

Every entry here is a failure that produces no error anywhere. That is why the health check exists.

More Docs

• README.md — the project, its staging, and the design reasoning
• AGENTS.md — engineering rules, including "never write Agora from memory"
• docs/AGORA-NOTES.md — Agora fields verified at runtime, with dates
• docs/VOBIZ.md — telephony fields verified at runtime, with dates
• docs/DEPLOY.md — why this needs a long-lived Node process, not serverless
• kaun-katega-taarpati-spec.md — the game design spec