Speaker Lock

# Agora Conversational AI — Voiceprint (Speaker Lock) Recipe (Python)

[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](./LICENSE)
[![Python](https://img.shields.io/badge/python-%3E%3D3.10-blue)](https://www.python.org/)
[![Bun](https://img.shields.io/badge/bun-latest-black)](https://bun.sh/)

The **voiceprint / speaker-lock** recipe in the Agora Conversational AI recipes family.
The agent auto-locks onto the primary speaker and suppresses other voices and background
noise using Agora **Speaker Lock** (`sal_mode: "locking"`). Fully **zero-key** — no
voiceprint enrollment required, and OpenAI is Agora-managed (no `OPENAI_API_KEY` needed
unless you bring your own account).

**Pipeline:** `DeepgramSTT(nova-3)` → `OpenAI` (plain assistant) → `MiniMaxTTS`

**Speaker Lock** (`sal_mode: "locking"`) — the SDK auto-locks onto the first clear
speaker detected in the channel and suppresses all other voices and background noise.
No enrollment step, no voiceprint file required.

> **Optional named-speaker mode (not used here):** The SDK also supports
> `sal_mode: "recognition"`, which locks onto a *specific* named speaker. This mode
> requires a pre-hosted 16 kHz / 16-bit mono PCM voiceprint (≤ 2 MB) supplied via
> `sample_urls`. The SDK has no voiceprint-enrollment API — you must host the PCM file
> yourself. This recipe uses only `"locking"` (zero-key, no enrollment).

## Prerequisites

- [Python 3.10+](https://www.python.org/)
- [Bun](https://bun.sh/)
- [Agora CLI](https://github.com/AgoraIO/cli) — makes generating an App ID + App Certificate easy

The same commands work on macOS, Linux, and Windows. On macOS/Linux, setup uses
`python3`; on Windows, it uses the Python launcher (`py`) or `python`. WSL and
virtualenv activation are not required.

## Run It

```bash
# 1. Install web deps + create the Python venv
bun run setup

# 2. Add Agora credentials (CLI), or edit server/.env.local by hand
agora login
agora project use <your-project>          # select which project to use
agora project env write server/.env.local # writes App ID + Certificate

# 3. Run backend + web
bun run dev
```

Open [http://localhost:3000](http://localhost:3000) → **Start Conversation** → speak.

### Working from a clone

If you cloned this repo (rather than scaffolding via the Agora CLI), the steps
above are complete as written: `bun run setup` creates the Python venv and
installs web dependencies, then `bun run dev` brings up both services. You
still need Agora credentials in `server/.env.local` before a conversation can connect.

Services:

- Frontend — http://localhost:3000
- Backend — http://localhost:8000
- Mock LLM — N/A (managed OpenAI, no local service)
- API docs — http://localhost:8000/docs

## Deploy

Deploy `web` (Next.js) and `server` (a reachable FastAPI backend). Set
`AGENT_BACKEND_URL` in the web deployment so the Next rewrites reach the backend.

A backend-only Docker image is published to
`ghcr.io/AgoraIO-Conversational-AI/recipe-agent-voiceprint` on `v*` tags.
It exposes **BACKEND-ONLY** (:8000). No separate LLM container is needed —
OpenAI is Agora-managed.

## Environment variables

Backend env file: [`server/.env.example`](https://github.com/AgoraIO-Conversational-AI/recipe-agent-voiceprint/blob/main/server/.env.example).

| Variable | Required | Default | Notes |
| --- | :---: | :---: | --- |
| `AGORA_APP_ID` | ✅ | — | Agora Console → Project → App ID |
| `AGORA_APP_CERTIFICATE` | ✅ | — | Agora Console → Project → App Certificate |
| `OPENAI_MODEL` | | `gpt-4o-mini` | OpenAI model |
| `OPENAI_API_KEY` | | — | Optional — Agora manages the OpenAI key by default (keyless). Set only if your account requires it. |
| `TTS_VOICE` | | `English_captivating_female1` | MiniMax TTS voice |
| `AGENT_GREETING` | | built-in | Optional opening line override |

## Commands

```bash
bun run setup            # install web deps + create server/ venv
bun run dev              # run backend (:8000) + web (:3000)

bun run doctor           # prerequisite check (no creds needed)
bun run doctor:local     # + .env.local + credentials checks

bun run verify           # web-only gate (no Agora creds needed)
bun run verify:local     # full local gate: backend compile + smoke tests + web build
bun run clean            # remove venvs and build artifacts
```

Tests run standalone (no Agora cloud needed): `pytest` in `server/`, plus
`bun run verify` in `web/`. CI runs them on Linux/macOS/Windows × Python 3.10 & 3.13.

## Architecture

```
Browser (localhost:3000)
  │  fetch /api/*
  ▼
Next.js  ──rewrite──▶  Agent backend  (server/, localhost:8000)
                          │  starts agent session (managed OpenAI vendor)
                          │  sal={"sal_mode": "locking"}  ← Speaker Lock
                          ▼
                       Agora ConvoAI Cloud
                          │  Deepgram STT (managed, nova-3)
                          │  Speaker Lock — locks onto primary speaker, suppresses others
                          │  OpenAI assistant (Agora-managed, keyless)
                          │  MiniMax TTS (managed)
                          ▼
                       User hears agent focused on their voice only
```

No separate `llm/` service — OpenAI is Agora-managed and requires no API key.
See [ARCHITECTURE.md](https://github.com/AgoraIO-Conversational-AI/recipe-agent-voiceprint/blob/main/ARCHITECTURE.md).

## What You Get

- A **Next.js** web client (:3000) that drives the RTC/RTM lifecycle and only ever calls `/api/*`.
- A **FastAPI** agent backend (:8000) that owns Agora token generation and the agent session lifecycle.
- The `/api/get_config` · `/api/startAgent` · `/api/stopAgent` contract between the web client and the backend (Next rewrites, no Route Handlers).
- **Speaker Lock** (`sal_mode: "locking"`) wired via `sal=build_sal()` on `AgoraAgent` — no enrollment, no extra credentials.
- **Managed keyless OpenAI** as a plain conversational assistant — Agora-managed, no `OPENAI_API_KEY` required.
- **Zero-key** setup — the full pipeline runs with only Agora credentials.

## How It Works

1. The browser calls `/api/get_config`, which Next rewrites to the backend; the
   backend mints an Agora token from `AGORA_APP_ID` + `AGORA_APP_CERTIFICATE`.
2. The browser joins the RTC channel, then calls `/api/startAgent`; the backend
   starts an agent session with `sal={"sal_mode": "locking"}` on `AgoraAgent`.
3. Agora's Speaker Lock detects the first clear speaker in the channel and locks onto
   that voice, suppressing other voices and background noise automatically.
4. Deepgram STT transcribes the locked speaker's audio.
5. Agora's managed OpenAI stage replies with a concise assistant response.
6. MiniMax TTS speaks the response back into the channel.
7. `/api/stopAgent` ends the session.

## Repo Map

- `web/` — Next.js frontend (:3000); RTC/RTM lifecycle and UI.
- `server/` — FastAPI agent backend (:8000); Agora tokens + agent lifecycle.
- `server/src/sal_config.py` — pure builder for the SAL (Speaker Lock) config dict.
- `ARCHITECTURE.md` — system shape and component boundaries.
- `AGENTS.md` — guide for coding agents working in this repo.

## Troubleshooting

| Problem | Fix |
| --- | --- |
| Agent does not lock onto my voice | Ensure only one speaker is active at the start; Speaker Lock latches onto the first clear voice. |
| Local calls fail under a global proxy (Clash, etc.) | Configure your proxy to send `127.0.0.1`, `localhost`, and RFC-1918 ranges DIRECT. |

## More Docs

- [ARCHITECTURE.md](https://github.com/AgoraIO-Conversational-AI/recipe-agent-voiceprint/blob/main/ARCHITECTURE.md)
- [AGENTS.md](https://github.com/AgoraIO-Conversational-AI/recipe-agent-voiceprint/blob/main/AGENTS.md)

## License

Released under the [MIT License](https://github.com/AgoraIO-Conversational-AI/recipe-agent-voiceprint/blob/main/LICENSE).