Recipes
投稿
開始使用
返回內容列表

Muse AI

使用 Agora RTC 構建實時語音 AI 音樂製作夥伴，協助旋律構思。

TypeScript / React進階Molly Arora
GitHub

實踐教程

本教程目錄

Muse AI - Conversational Voice Co-Producer & Studio Agent

Build an intelligent, real-time conversational Voice AI music studio assistant and melody ideation co-producer using Agora RTC and Agora Conversational AI.

Agora RTC Agora Conversational AI TypeScript React Vite

Overview

Muse AI is a real-time conversational Voice AI studio co-producer designed for musicians, songwriters, producers, and creative builders. When musical inspiration strikes — while humming a tune on the go, brainstorming lyrics, or crafting a chord progression — creators often struggle with the friction of configuring traditional Digital Audio Workstations (DAWs).

Muse AI bridges this gap with Agora's sub-second Voice AI capabilities:

  • Speak or Sing Directly: Users hum melodies or converse naturally with specialized AI agent personas.
  • Ultra-Low Latency Audio: Two-way conversational audio stream powered by the Agora RTC Web SDK (agora-rtc-sdk-ng).
  • Live Pitch Telemetry & Sing-to-Create: Real-time pitch tracking, fundamental frequency extraction, and automatic scale/key identification.
  • Dynamic Prompt Injection: Live steering of AI agent personality, speed, pitch, and domain context during active voice sessions.
  • In-Browser Web Audio Sandbox: Built-in polyphonic synthesizer and beat engine for rapid playback and acoustic prototyping.

Key Features

  1. Real-Time Voice AI Streaming with Agora RTC

    • High-fidelity stereo audio (high_quality_stereo profile).
    • Built-in Acoustic Echo Cancellation (AEC), Automatic Gain Control (AGC), and Acoustic Noise Suppression (ANS).
    • Real-time volume-indicator telemetry for dual-channel speaking feedback (user vs. AI agent).

  2. Agora Conversational AI REST Gateway

    • Direct session lifecycle management (/start, /stop).
    • Dynamic prompt injection via /sessions/{agentId}/inject-prompt.
    • Custom voice profile configuration (voice_id, temperature, speech_rate, pitch).

  3. Multi-Agent Creative Archetypes

    • Nova (Executive Strategist): Fast-paced decision matrices and release roadmaps.
    • Devon (Systems & Code Architect): Technical architecture, Web Audio DSP, and API integration.
    • Aria (Creative Director): Songwriting hooks, emotional arc planning, and lyric generation.
    • Sora (Multilingual Polyglot): Cross-lingual lyrics, phonetic coaching in 30+ languages.
    • Zenith (Vocal & Mindfulness Coach): Vocal warmups, breath telemetry, and performance coaching.

  4. Sing-to-Create & Audio Spectrum Analysis

    • High-speed pitch detection via Autocorrelation algorithm.
    • Live holographic visualizer and audio frequency spectrum.
    • Automatic scale and note mapping (e.g., C Major, A Minor).

  5. Lyrics Studio & Multitrack Export

    • Verse, chorus, and bridge structuring with real-time AI suggestions.
    • Session telemetry, transcription logs, and audio export (JSON / WAV / MIDI-compatible notes).

Architecture & Data Flow

text
+-------------------------------------------------------------------------+
|                               Browser Client                            |
|                                                                         |
|  +-------------------+      +------------------+     +---------------+  |
|  | User Microphone   | ---> | Agora RTC Track  | --> | Agora Channel |  |
|  +-------------------+      +------------------+     +-------+-------+  |
|           |                                                  |          |
|           v                                                  v          |
|  +-------------------+                             +-----------------+  |
|  | Web Audio Analyzer|                             | Agora Speaker   |  |
|  | (Pitch & Spectrum)|                             | Audio Track     |  |
|  +-------------------+                             +-----------------+  |
+--------------------------------------------------------------+----------+
                                                               |
                                            Agora Voice Network
                                                               |
                                                               v
                                              +---------------------------------+
                                              | Agora Conversational AI Agent   |
                                              | (LLM + TTS + STT Pipeline)      |
                                              +---------------------------------+

Prerequisites

  • Node.js: Version 18.0 or higher
  • npm: Version 9.0 or higher
  • Agora Developer Account:
    • Sign up at Agora Console.
    • Create a project to obtain an App ID and App Certificate / Token.
    • Enable Conversational AI in your Agora Console project settings.

Quickstart Guide

1. Clone the Repository

bash
git clone https://github.com/Molly-Arora02/MUSE-AI-.git
cd MUSE-AI-

2. Install Dependencies

bash
npm install

3. Start Development Server

bash
npm run dev

Open http://localhost:5173 in your browser.

4. Connect with Agora Voice AI

  1. Click "Start Creating" to enter the Studio.
  2. Click the Agora Settings gear icon in the top header.
  3. Enter your Agora credentials:
    • Agora App ID: your_agora_app_id
    • Channel Name: muse-studio-01
    • Token (Optional for testing, required for secured channels): your_rtc_token
    • Agent ID: agora-agent-vocalis-01
  4. Click "Save Settings", then toggle "Connect Voice" to start real-time conversation!

Demo / Offline Mode: If you do not have Agora credentials immediately available, click "Try Interactive Demo" on the landing page. Muse AI will run in client-side Web Audio synthesis mode with full interactive voice and pitch simulation.

Code Implementation Walkthrough

1. Initializing Agora RTC Client & Microphone Audio Track

In src/services/agoraService.ts:

typescript
import AgoraRTC from 'agora-rtc-sdk-ng';
import type { IAgoraRTCClient, IMicrophoneAudioTrack, IRemoteAudioTrack } from 'agora-rtc-sdk-ng';

export class AgoraRTCService {
  private client: IAgoraRTCClient | null = null;
  private localAudioTrack: IMicrophoneAudioTrack | null = null;
  private remoteAudioTrack: IRemoteAudioTrack | null = null;

  public async initializeAndJoin(config: AgoraConfig, callbacks: AgoraCallbacks): Promise<boolean> {
    this.client = AgoraRTC.createClient({ mode: 'rtc', codec: 'vp8' });

    // Handle remote AI agent voice track
    this.client.on('user-published', async (user, mediaType) => {
      if (mediaType === 'audio') {
        await this.client?.subscribe(user, mediaType);
        this.remoteAudioTrack = user.audioTrack || null;
        if (this.remoteAudioTrack) {
          this.remoteAudioTrack.play();
          callbacks.onAgentJoined?.(user.uid);
        }
      }
    });

    // Volume level indicator for dual speaking detection
    this.client.enableAudioVolumeIndicator();
    this.client.on('volume-indicator', (volumes) => {
      volumes.forEach((vol) => {
        const level = Math.min(100, Math.round((vol.level / 100) * 100));
        if (vol.uid === 0 || vol.uid === config.uid) {
          callbacks.onUserAudioLevel?.(level);
          callbacks.onUserSpeaking?.(level > 15);
        } else {
          callbacks.onAgentAudioLevel?.(level);
          callbacks.onAgentSpeaking?.(level > 15);
        }
      });
    });

    // Create microphone track with noise suppression
    this.localAudioTrack = await AgoraRTC.createMicrophoneAudioTrack({
      AEC: true,
      ANS: config.enableNoiseSuppression,
      AGC: true,
    });

    // Join RTC channel and publish microphone audio
    await this.client.join(config.appId, config.channel, config.token || null, config.uid);
    await this.client.publish([this.localAudioTrack]);
    return true;
  }
}

2. Starting Conversational AI Agent via Gateway

In src/services/agentApi.ts:

typescript
export class AgoraAgentGateway {
  public async startAgent(config: AgoraConfig, agent: VoiceAgent) {
    const endpoint = config.gatewayUrl.replace('{appId}', config.appId) + '/start';
    const payload = {
      name: agent.name,
      properties: {
        channel_name: config.channel,
        agent_rtc_uid: 'agent_' + config.agentId,
        remote_rtc_uids: [config.uid],
        voice_id: agent.voiceId,
        system_instruction: agent.systemPrompt,
        temperature: agent.temperature,
        speech_rate: agent.speed,
        pitch: agent.pitch,
        tools: agent.tools
      }
    };

    const res = await fetch(endpoint, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Authorization': `Bearer ${config.token || 'demo-token'}`
      },
      body: JSON.stringify(payload)
    });

    return await res.json();
  }

  public async injectPrompt(config: AgoraConfig, agentId: string, prompt: string) {
    const endpoint = config.gatewayUrl.replace('{appId}', config.appId) + `/sessions/${agentId}/inject-prompt`;
    return await fetch(endpoint, {
      method: 'POST',
      headers: { 'Content-Type': 'application/json' },
      body: JSON.stringify({ prompt, timestamp: Date.now() })
    });
  }
}

Configuration Reference

OptionTypeDefaultDescription
appIdstring""Agora App ID from Agora Console
channelstring"vocalis-demo-room"Target RTC channel name
tokenstring""Temporary or server-generated RTC token
uidnumberRandom 6-digitUnique client numeric ID
enableNoiseSuppressionbooleantrueEnables Agora hardware/software noise filtering
audioProfilestring"high_quality_stereo"Agora RTC audio profile
vadSensitivitynumber75Voice Activity Detection sensitivity

Tech Stack

  • Frontend: React 19, TypeScript, Tailwind CSS
  • Voice & Real-Time: Agora RTC SDK (agora-rtc-sdk-ng v4.24+)
  • Audio DSP: Web Audio API (AnalyserNode, BiquadFilter, Custom Oscillators)
  • Icons & Visuals: Lucide React, Canvas Confetti
  • Build Tool: Vite 8.2

License

This project is licensed under the MIT License. See LICENSE for details.