Muse AI - Conversational Voice Co-Producer & Studio Agent

Build an intelligent, real-time conversational Voice AI music studio assistant and melody ideation co-producer using Agora RTC and Agora Conversational AI.

    

Overview

Muse AI is a real-time conversational Voice AI studio co-producer designed for musicians, songwriters, producers, and creative builders. When musical inspiration strikes — while humming a tune on the go, brainstorming lyrics, or crafting a chord progression — creators often struggle with the friction of configuring traditional Digital Audio Workstations (DAWs).

Muse AI bridges this gap with Agora's sub-second Voice AI capabilities:

• Speak or Sing Directly: Users hum melodies or converse naturally with specialized AI agent personas.
• Ultra-Low Latency Audio: Two-way conversational audio stream powered by the Agora RTC Web SDK (agora-rtc-sdk-ng).
• Live Pitch Telemetry & Sing-to-Create: Real-time pitch tracking, fundamental frequency extraction, and automatic scale/key identification.
• Dynamic Prompt Injection: Live steering of AI agent personality, speed, pitch, and domain context during active voice sessions.
• In-Browser Web Audio Sandbox: Built-in polyphonic synthesizer and beat engine for rapid playback and acoustic prototyping.

Key Features

1. Real-Time Voice AI Streaming with Agora RTC

• High-fidelity stereo audio (high_quality_stereo profile).
• Built-in Acoustic Echo Cancellation (AEC), Automatic Gain Control (AGC), and Acoustic Noise Suppression (ANS).
• Real-time volume-indicator telemetry for dual-channel speaking feedback (user vs. AI agent).

1. Agora Conversational AI REST Gateway

• Direct session lifecycle management (/start, /stop).
• Dynamic prompt injection via /sessions/{agentId}/inject-prompt.
• Custom voice profile configuration (voice_id, temperature, speech_rate, pitch).

1. Multi-Agent Creative Archetypes

• Nova (Executive Strategist): Fast-paced decision matrices and release roadmaps.
• Devon (Systems & Code Architect): Technical architecture, Web Audio DSP, and API integration.
• Aria (Creative Director): Songwriting hooks, emotional arc planning, and lyric generation.
• Sora (Multilingual Polyglot): Cross-lingual lyrics, phonetic coaching in 30+ languages.
• Zenith (Vocal & Mindfulness Coach): Vocal warmups, breath telemetry, and performance coaching.

1. Sing-to-Create & Audio Spectrum Analysis

• High-speed pitch detection via Autocorrelation algorithm.
• Live holographic visualizer and audio frequency spectrum.
• Automatic scale and note mapping (e.g., C Major, A Minor).

1. Lyrics Studio & Multitrack Export

• Verse, chorus, and bridge structuring with real-time AI suggestions.
• Session telemetry, transcription logs, and audio export (JSON / WAV / MIDI-compatible notes).

Architecture & Data Flow

+-------------------------------------------------------------------------+
|                               Browser Client                            |
|                                                                         |
|  +-------------------+      +------------------+     +---------------+  |
|  | User Microphone   | ---> | Agora RTC Track  | --> | Agora Channel |  |
|  +-------------------+      +------------------+     +-------+-------+  |
|           |                                                  |          |
|           v                                                  v          |
|  +-------------------+                             +-----------------+  |
|  | Web Audio Analyzer|                             | Agora Speaker   |  |
|  | (Pitch & Spectrum)|                             | Audio Track     |  |
|  +-------------------+                             +-----------------+  |
+--------------------------------------------------------------+----------+
                                                               |
                                            Agora Voice Network
                                                               |
                                                               v
                                              +---------------------------------+
                                              | Agora Conversational AI Agent   |
                                              | (LLM + TTS + STT Pipeline)      |
                                              +---------------------------------+

Prerequisites

• Node.js: Version 18.0 or higher
• npm: Version 9.0 or higher
• Agora Developer Account:
• Sign up at Agora Console.
• Create a project to obtain an App ID and App Certificate / Token.
• Enable Conversational AI in your Agora Console project settings.

Quickstart Guide

1. Clone the Repository

git clone https://github.com/Molly-Arora02/MUSE-AI-.git
cd MUSE-AI-

2. Install Dependencies

npm install

3. Start Development Server

npm run dev

Open http://localhost:5173 in your browser.

4. Connect with Agora Voice AI

1. Click "Start Creating" to enter the Studio.
2. Click the Agora Settings gear icon in the top header.
3. Enter your Agora credentials:

• Agora App ID: your_agora_app_id
• Channel Name: muse-studio-01
• Token (Optional for testing, required for secured channels): your_rtc_token
• Agent ID: agora-agent-vocalis-01

1. Click "Save Settings", then toggle "Connect Voice" to start real-time conversation!

Demo / Offline Mode: If you do not have Agora credentials immediately available, click "Try Interactive Demo" on the landing page. Muse AI will run in client-side Web Audio synthesis mode with full interactive voice and pitch simulation.

Code Implementation Walkthrough

1. Initializing Agora RTC Client & Microphone Audio Track

In src/services/agoraService.ts:

import AgoraRTC from 'agora-rtc-sdk-ng';
import type { IAgoraRTCClient, IMicrophoneAudioTrack, IRemoteAudioTrack } from 'agora-rtc-sdk-ng';

export class AgoraRTCService {
  private client: IAgoraRTCClient | null = null;
  private localAudioTrack: IMicrophoneAudioTrack | null = null;
  private remoteAudioTrack: IRemoteAudioTrack | null = null;

  public async initializeAndJoin(config: AgoraConfig, callbacks: AgoraCallbacks): Promise<boolean> {
    this.client = AgoraRTC.createClient({ mode: 'rtc', codec: 'vp8' });

    // Handle remote AI agent voice track
    this.client.on('user-published', async (user, mediaType) => {
      if (mediaType === 'audio') {
        await this.client?.subscribe(user, mediaType);
        this.remoteAudioTrack = user.audioTrack || null;
        if (this.remoteAudioTrack) {
          this.remoteAudioTrack.play();
          callbacks.onAgentJoined?.(user.uid);
        }
      }
    });

    // Volume level indicator for dual speaking detection
    this.client.enableAudioVolumeIndicator();
    this.client.on('volume-indicator', (volumes) => {
      volumes.forEach((vol) => {
        const level = Math.min(100, Math.round((vol.level / 100) * 100));
        if (vol.uid === 0 || vol.uid === config.uid) {
          callbacks.onUserAudioLevel?.(level);
          callbacks.onUserSpeaking?.(level > 15);
        } else {
          callbacks.onAgentAudioLevel?.(level);
          callbacks.onAgentSpeaking?.(level > 15);
        }
      });
    });

    // Create microphone track with noise suppression
    this.localAudioTrack = await AgoraRTC.createMicrophoneAudioTrack({
      AEC: true,
      ANS: config.enableNoiseSuppression,
      AGC: true,
    });

    // Join RTC channel and publish microphone audio
    await this.client.join(config.appId, config.channel, config.token || null, config.uid);
    await this.client.publish([this.localAudioTrack]);
    return true;
  }
}

2. Starting Conversational AI Agent via Gateway

In src/services/agentApi.ts:

export class AgoraAgentGateway {
  public async startAgent(config: AgoraConfig, agent: VoiceAgent) {
    const endpoint = config.gatewayUrl.replace('{appId}', config.appId) + '/start';
    const payload = {
      name: agent.name,
      properties: {
        channel_name: config.channel,
        agent_rtc_uid: 'agent_' + config.agentId,
        remote_rtc_uids: [config.uid],
        voice_id: agent.voiceId,
        system_instruction: agent.systemPrompt,
        temperature: agent.temperature,
        speech_rate: agent.speed,
        pitch: agent.pitch,
        tools: agent.tools
      }
    };

    const res = await fetch(endpoint, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${config.token || 'demo-token'}`
      },
      body: JSON.stringify(payload)
    });

    return await res.json();
  }

  public async injectPrompt(config: AgoraConfig, agentId: string, prompt: string) {
    const endpoint = config.gatewayUrl.replace('{appId}', config.appId) + `/sessions/${agentId}/inject-prompt`;
    return await fetch(endpoint, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ prompt, timestamp: Date.now() })
    });
  }
}

Configuration Reference

| Option | Type | Default | Description | | :--- | :--- | :--- | :--- | | appId | string | "" | Agora App ID from Agora Console | | channel | string | "vocalis-demo-room" | Target RTC channel name | | token | string | "" | Temporary or server-generated RTC token | | uid | number | Random 6-digit | Unique client numeric ID | | enableNoiseSuppression| boolean | true | Enables Agora hardware/software noise filtering | | audioProfile | string | "high_quality_stereo" | Agora RTC audio profile | | vadSensitivity | number | 75 | Voice Activity Detection sensitivity |

Tech Stack

• Frontend: React 19, TypeScript, Tailwind CSS
• Voice & Real-Time: Agora RTC SDK (agora-rtc-sdk-ng v4.24+)
• Audio DSP: Web Audio API (AnalyserNode, BiquadFilter, Custom Oscillators)
• Icons & Visuals: Lucide React, Canvas Confetti
• Build Tool: Vite 8.2

License

This project is licensed under the MIT License. See LICENSE for details.