Recipes
Contribuir
Empezar
Volver a Recipes

Aria — Real-Time Voice AI Sales Agent

Un agente de ventas por voz que negocia y actualiza el CRM durante la llamada.

Python / TypeScriptAvanzadoAryan-Techie
Probar demoGitHub
Llamada de Aria con transcripción y controles de llamada
Probar demo en vivo

Necesita micrófono

Guía de implementación

En esta guía
Aria logo

ARIA

A real-time voice AI that sells like a real salesperson — not a phone tree.

Built with Agora Backend Frontend Mobile Contributions welcome

Part of the AROICE family of tools.

Features • Quick Start • Architecture • Tech Stack • Contributing • Team

🔴 Live now: aria.aroice.in — console + ops dashboard · crm.aria.aroice.in — the real CRM, hosted on a real Oracle Cloud VPS behind Caddy and Let's Encrypt TLS, not a laptop and a tunnel.

Aria console mid-call — the lead card, sentiment, and live deal state updating while she is still talking

Aria runs a complete B2B sales qualification call, out loud, in real time. She answers pricing questions from a knowledge base, handles objections, negotiates within limits she is not allowed to cross, remembers what you told her twenty turns ago, writes to a real CRM while you are still talking, books a meeting with a calendar invite, and escalates to a human — with a written handoff brief — the moment she genuinely should.

Open the console and a CRM lead page side by side. Say "actually, make that fifty devices" — the lead updates itself, with no refresh, while she is still speaking.

Aria console after the call ends — the AI-written brief, recommended next action, and handoff status

Contents

Features

  • Grounded, not scripted. No decision tree — an LLM-driven tool-calling loop decides, turn by turn, whether to search product docs, update the lead, check the calendar, negotiate, or escalate. Every pricing/spec answer comes from a live knowledge-base search; she says "I'm not sure" rather than invent a number.
  • A correction is a field update, not a restart. Change "10 devices" to "50" mid-call and one field on one lead changes — you don't get a second lead for the same conversation.
  • A real CRM, live. EspoCRM in Docker backs every lead and meeting. Leads are created and updated mid-call, meetings are booked for real, and the CRM's own audit trail shows an agent genuinely working the record.
  • She bargains, and she can't give the deal away. A three-layer negotiation: Aria can offer up to 3% herself, a separate deal-desk agent can go to 10% against a real commitment, and nothing ever crosses an 18% walk-away floor — enforced by code, not by asking the model nicely.
  • A specialist for the technical questions. A second agent, its own prompt and its own voice, answers compatibility/migration/MDM/rollout questions from the real docs — and is required to say exactly what it doesn't know instead of rounding a gap to reassurance.
  • Escalation is a safety net, not just the model's opinion. Deterministic guardrails — repeated frustration, an unresolved objection raised three times, a low-confidence search — force a handoff even if the model never asks for one. Escalation is a warm transfer: a human joins the live call already briefed, and the customer never repeats themselves.
  • Booking sends a real invite. calendar_book_meeting emails the customer a confirmation with a .ics attachment that lands straight in their Google or Outlook calendar.
  • Every call ends with a human being told. Not just escalations — an AI-written wrap-up (headline, next action, what was already agreed) is delivered to the CRM, Slack, and email, independently and best-effort.
  • She can take the call in Hindi. One AGENT_LANGUAGE setting moves five things together — ASR, TTS voice, voice boosting, greeting, and prompt — so a Hinglish call doesn't half-switch.
  • The voice changes with who's actually talking. Aria, the deal desk, and the solutions engineer each get a distinct voice, and the accent follows the caller's language mid-call.
  • A mobile app for the rep. An Expo app that surfaces live escalations and handoff briefs, so the human on the other end of a warm transfer isn't stuck at a desktop.
  • Not just B2B. Aria reads the phrasing — "my team" vs. "I need a new phone" — and skips the company/role/headcount questions entirely for a solo buyer. Financing, AppleCare+, and trade-in work exactly the same for one device as for fifty.
  • A live ops dashboard. /dashboard lists every lead and session the backend has ever seen, live capacity stats, open escalations (with one-click discount approval), follow-up tasks, and the product catalog — proof concurrency is real, not simulated.
  • A catalog a rep can extend live. Adding a product writes real stock and re-indexes the RAG corpus in the same call — Aria can answer about it on the very next question, no restart.
  • When it isn't ready to be a meeting yet. A follow-up gets written as a real Task in the CRM, not a line buried in a transcript, so a rep has something to work later.
  • She does the currency math. ₹, rupees, lakh, crore — Aria recognizes all of them, converts at an approximate stated rate, and switches her own spoken currency to match the caller's.
  • A face, not just a voice. An optional 3D avatar, lip-synced off the live remote audio track in real time — toggle it on mid-call, the 2D orb stays the default.

Quick Start

Windows — one command:

bat
run.bat

That checks your prerequisites, starts EspoCRM in Docker, provisions it, launches the backend, opens a public tunnel, writes the tunnel URL into .env, restarts the backend so it picks the URL up, starts the frontend, and opens both tabs. run.bat stop shuts it all down.

Before the first run you need:

WhyGet it
Docker DesktopRuns EspoCRM + MariaDBhttps://docs.docker.com/desktop/install/windows-install/
Python 3.11+The backendhttps://www.python.org/downloads/
Node.js 18+The frontendhttps://nodejs.org/
cloudflaredPublic HTTPS URL so Agora can reach your machinewinget install --id Cloudflare.cloudflared
A .envCredentials — see belowcopy backend\.env.example .env

run.bat verifies all five and tells you exactly what's missing before it changes anything.

Minimum .env to make a call (at the repo root, not backend/):

ini
AGORA_APP_ID=...
AGORA_APP_CERTIFICATE=...
AGORA_CUSTOMER_KEY=...
AGORA_CUSTOMER_SECRET=...
ANTHROPIC_API_KEY=...
GROQ_API_KEY=...              # optional, but ~4x faster per hop
LLM_SHARED_SECRET=...         # any long random string — see Security

Then open http://localhost:3000 and click Start. (Not the tunnel URL — that serves the backend only.)

Manual startup, or on macOS/Linux
bash
# 1. CRM — first run pulls images and installs its own database (~2 min)
docker compose -f crm/docker-compose.yml up -d
python scripts/provision_crm.py          # role, API key, custom fields, layout
# paste the printed ESPOCRM_* lines into .env

# 2. backend
cd backend && python -m venv .venv && .venv/bin/pip install -r requirements.txt
.venv/bin/uvicorn app.main:app --port 8000

# 3. tunnel — then put the https URL into PUBLIC_BASE_URL and RESTART the backend
cloudflared tunnel --url http://localhost:8000

# 4. frontend
cd frontend && npm install && npm run dev

Settings are cached with lru_cache, so editing .env while the backend runs does nothing until it's restarted. Quick-tunnel hostnames change on every cloudflared start.

Architecture

text
browser (Next.js :3000) ──RTC audio──► Agora Conversational AI Engine
                                         ASR Deepgram → LLM → TTS MiniMax
                                                │
                       POST {PUBLIC_BASE_URL}/agent/{session}/v1/chat/completions
                                                │
                                    FastAPI (:8000) ── this repo
                                      ├─ Anthropic primary / Groq for speed
                                      ├─ tool loop (10 tools, max 6 hops)
                                      ├─ deal desk + solutions engineer (layer 2)
                                      ├─ RAG over the product docs
                                      ├─ CRM + calendar ──► EspoCRM (:8080, Docker)
                                      ├─ confirmation email + .ics invite ──► SMTP
                                      ├─ escalation inbox ──► Slack + mobile app
                                      └─ SSE stream back to Agora

Agora is the phone system. This backend is the person. Agora owns WebRTC transport, the Deepgram/MiniMax connections, voice activity detection, turn-taking, and barge-in — and calls this backend as if it were OpenAI, streaming the response into TTS chunk by chunk. This repo owns the persona, the tool loop, RAG, the CRM, qualification state, negotiation, escalation, and every word she actually says. Agora never sees any of it.

The diagram above is local dev — aria.aroice.in runs the identical stack on a real Oracle Cloud VPS behind Caddy, one hostname split by path so the browser never makes a cross-origin request. The backend runs --workers 1 there, deliberately: sessions live in an in-process dict, so a second worker would get its own empty copy and Agora's next webhook for a call would land on a process that never heard of it. Scaling out means moving sessions to Redis first, not adding workers.

Tech Stack

LayerWhatWhy
Agora Conversational AI EngineReal-time voice cloud (RTC audio + RTM data channel)The whole phone-call layer — ASR, barge-in, TTS — without building real-time audio ourselves
Deepgram Nova-3Speech-to-text, multi-language modeHandles Hindi/English mixed in one sentence (Hinglish)
MiniMax speech-2.8-turboText-to-speech, three distinct voicesAria, the deal desk, and the solutions engineer each sound like a different person
Anthropic Claude / GroqThe reasoning model, run on one turnGroq for speed, Anthropic as the quality fallback — chosen by measurement, not guesswork
FastAPIPython backendEvery tool, every rule, every word Aria says
Next.js + ReactOperator consoleLive transcript, lead card, signals, deal, handoff
Expo / React NativeMobile app for the repEscalations and handoff briefs on the move
EspoCRMOpen-source CRM, in DockerThe real system of record — leads and meetings, not a mock
mem0 + Voyage AILong-term call memoryRecalls something said early in a long call
Three.jsOptional 3D avatarReal lipsync off live audio, plain WebGL — no react-three-fiber
cloudflaredCloudflare TunnelPublic HTTPS so Agora's cloud can reach a laptop during local dev
Caddy + Let's EncryptReverse proxy, automatic TLSThe live deployment — one hostname split by path, real HTTPS that renews itself
Oracle Cloud (ARM/A1.Flex)The VPS aria.aroice.in runs onHosts console, backend, and CRM behind Caddy, --workers 1 by design (see Architecture)
SMTPEmail deliveryConfirmation invites and end-of-call wrap-ups

How a call actually flows

  1. The customer speaks in the browser; audio goes to Agora over RTC.
  2. Agora runs Deepgram ASR and gets text.
  3. Agora sends that text to this backend, in the same shape as an OpenAI chat request.
  4. The backend builds the prompt: Aria's persona, the current lead state, memory recalled from mem0, and the tool list.
  5. The model decides: answer directly, or call a tool first — search products, update the lead, log an objection, negotiate, ask the solutions engineer, check the calendar, book a meeting, or escalate.
  6. The tool runs, the result feeds back, and the model writes the reply.
  7. The reply streams back to Agora token by token, so TTS starts on the first chunk.
  8. Every tool call and transcript turn is written to the session's event log, which the console polls and renders live.

Three layers of who is talking

  • Layer 1 — Aria. The only one who talks to the customer. Warm, generalist, holds the conversation and can offer up to 3% off on her own.
  • Layer 2 — two specialists. Neither talks to the customer directly; Aria relays the answer, in a distinct voice.
    • Deal desk — reasons about margin. Can go to 10%, only against a real commitment, and every round moves less than the last so the floor is felt coming.
    • Solutions engineer — handles compatibility, migration, MDM, and rollout questions from the real docs, and is required to state exactly what it doesn't know.
  • Layer 3 — a human. Reached by escalation, as a warm transfer: they join the live call already knowing the context, and the customer never repeats themselves.

The console

  • Orb + caption — who is speaking, driven by real audio levels.
  • Transcript — customer and Aria turns, streamed live as words are generated.
  • Tool calls — every tool fired, with its latency in milliseconds.
  • Lead — company, devices, budget, timeline, and the qualification stage.
  • Signals — sentiment right now.
  • Deal — the three limits (Aria/desk/floor) and every round offered.
  • Handoff — the escalation guardrails filling up live, then the join link.
  • Brief — after the call ends, the AI-written wrap-up for the rep.

The CRM

Not a mock — a real EspoCRM instance, running in Docker, backing every product, lead, and meeting Aria touches.

EspoCRM Products list — the real catalog Aria's pricing tool searches EspoCRM Product detail view — an iPhone 15 record

scripts/provision_crm.py sets it up headlessly — role, API key, custom fields, and layout — so it's reproducible on a fresh machine instead of twenty minutes of clicking.

Mobile app for the rep

A warm transfer needs somewhere for the human to land. The Expo app surfaces the live escalation queue and each call's AI-written brief, so a rep isn't stuck at a desktop waiting for a handoff.

Aria mobile app — escalation inbox showing a customer waiting and the AI-written call outcome

The Configuration

Everything is environment-driven — see backend/.env.example for the full annotated list.

VariablePurpose
PUBLIC_BASE_URLPublic HTTPS URL Agora calls back into — changes on every tunnel restart
LLM_SHARED_SECRETRequired — the webhook is public; this is what stops a stranger using your LLM bill
LLM_PROVIDERgroq (default, fast) or anthropic
CRM_BACKENDespocrm or memory (automatic fallback if EspoCRM is unreachable)
EMAIL_ENABLEDConfirmation email + .ics invite on booking — off by default
AGENT_LANGUAGEen, hi, or hinglish — moves ASR, voice, boost, greeting and prompt together
VOICE_SWITCHING_ENABLEDFollow the caller's language and give each layer its own voice

Tests

bash
cd backend && python -m pytest

293 tests, ~2 seconds, zero network calls — the full tool loop runs against fake models and fake stores: corrections, escalation rules, deal clamping, voice selection, and the LLM endpoint.

Proven at scale

scripts/capacity_test.py runs the real six-beat call script — CRM writes, deal-desk consult, RAG search, calendar lookup — through the real turn loop, concurrently. Only the model itself is stubbed.

Concurrent calls256
Turns handled1,536 in 1.91s
p95 latency0.30s
Failed turns0

At 512 concurrent calls: 3,072 turns in 6.34s (485/s), p95 1.04s, still zero failed. It found a genuine concurrency bug on its first serious run — before that bug ever reached a live call.

Project Layout

PathWhat
backend/app/orchestrator/pipeline.pyThe turn loop — run_turn_stream is the production path
backend/app/deal/The negotiation engine: policy, clamp, and the deal-desk agent
backend/app/specialists/solutions.pyThe solutions engineer, and what it refuses to claim
backend/app/crm/ · backend/app/calendar/EspoCRM leads and meetings
backend/app/notify/Confirmation email, .ics invite, and the end-of-call wrap-up
backend/app/language/profiles.pyEnglish / Hindi / Hinglish, and the settings that must agree
backend/app/voice/director.pyWho speaks, in what accent, and when it's worth a round-trip
backend/app/tasks/Follow-up tasks — written as real EspoCRM Tasks, not transcript notes
frontend/app/page.tsxThe Next.js operator console
frontend/app/dashboard/The ops dashboard — leads, capacity, escalations, catalog, in one view
frontend/app/about/The project's own story/architecture page
frontend/components/Avatar3D.tsx · lib/lipsync.tsThe optional 3D avatar and its real-time lipsync
mobile/The Expo app for the rep
crm/EspoCRM + MariaDB Docker setup, and scripts/provision_crm.py
deploy/The live-VPS deployment — Caddy, docker-compose, up.sh

Contributing

Contributions are welcome. Open an issue for a bug or a feature idea, or send a pull request — please describe what changed and why in the PR description.

Team

Built by Team AR Voice, under AROICE.

Aryan Techie (Aryan Jangra)GitHub · LinkedIn
Rudra Pratap SinghGitHub
Shipra PorwalLinkedIn

Made with ❤️ by AROICE