Recipes
All recipes
AndroidPythonIntermediate

Android AutoPilot Voice Action Agent

Control Android apps hands-free using Agora Conversational AI and Gemini Vision.

Source on GitHubView raw recipe

Use with Agora CLI

Clone the recipe and configure it with an Agora project.

Agora CLI
agora init my-android-autopilot-voice-agent --recipe android-autopilot-voice-agent

Recipe prompt

Paste into Cursor, Claude Code, v0, or your coding agent
Use with your coding agent
You are implementing the "Android AutoPilot Voice Action Agent" recipe in this project.

Read the recipe markdown first:
https://raw.githubusercontent.com/Shariq1912/AgoraActionAgent/main/RECIPE.md

Use the source repository for cross-reference:
https://github.com/Shariq1912/AgoraActionAgent

Build this recipe into the user's app using the markdown as the implementation guide. Inspect related source files through the repository links when the recipe points to them. Ask before installing new dependencies.

Recipe

Rendered from the configured recipe markdown.

Raw

Android AutoPilot Voice Action Agent

An autonomous voice assistant for Android built with Agora Conversational AI and Gemini Vision. It listens to natural voice instructions in real time and executes hands-free UI actions (tap, swipe, type, dismiss recents, scroll wheels, set alarms) directly on the device using Android's Accessibility Service.

Key Features

  • Real-time Conversational Voice: Powered by Agora Conversational AI, Deepgram ASR (en-IN Indian English), and MiniMax TTS.
  • Natural Intent Recognition: Speaks friendly confirmations aloud before initiating tasks without reading out code or raw JSON.
  • Autonomous Vision & UI Execution: Uses Gemini Vision to parse Android UI accessibility layouts and dynamically calculate touch coordinates and gestures.
  • App Control Capabilities:
  • Open & close apps
  • Dismiss specific app cards from Recents (smart gesture pathing)
  • Control video playback (YouTube tap-to-reveal & pause)
  • Set alarm scroll wheels (NumberPicker)
  • Navigate Google Maps & send messages

System Architecture

User Voice Input
       │
       ▼
Agora Conversational AI (Cloud Agent)
 ├── Deepgram ASR (en-IN)
 ├── GPT-4o-mini LLM
 └── MiniMax TTS
       │
  RTM Action Tag: <action tool="agora_auto_pilot" command="..."/>
       │
       ▼
Android App (ActionAccessibilityService)
       │
  Screen Layout XML
       │
       ▼
Gemini Vision Model (Decide Next UI Action)
       │
       ▼
Android Accessibility API (Perform Tap / Swipe / Type)

Prerequisites

  • Android Studio Jellyfish or newer
  • Android Device running Android 10+ (API 29+)
  • Agora App ID & App Certificate
  • Python 3.10+ (for the backend proxy server)
  • Google Gemini API Key

Setup & Configuration

1. Backend Server Setup

cd server
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Create a .env file inside the server/ directory:

AGORA_APP_ID=your_agora_app_id
AGORA_APP_CERTIFICATE=your_agora_app_certificate
AGORA_AGENT_UID=123456
AGORA_AREA=NORTH_AMERICA
ASR_MODEL=nova-3
LLM_MODEL=gpt-4o-mini
TTS_MODEL=speech_2_6_turbo
TTS_VOICE_ID=English_captivating_female1
HOST=127.0.0.1
PORT=8000

Start the backend:

./run.sh

Expose the local server to the internet using localtunnel or ngrok:

npx localtunnel --port 8000

2. Android App Setup

  1. Open the project in Android Studio.
  2. In local.properties, set your localtunnel server URL:

``properties QUICKSTART_SERVER_URL=https://your-tunnel-url.loca.lt ``

  1. Build and install the app on your Android device.
  2. Grant Accessibility permissions:
  • Go to Settings > Accessibility > Installed Apps > Agora AutoPilot.
  • Turn ON accessibility access.

Usage

  1. Launch the app and tap Connect.
  2. Speak naturally to the agent:
  • "Open WhatsApp and text Mom"
  • "Pause the video on YouTube"
  • "Close Camera app from recent apps"
  • "Set an alarm for 5 AM tomorrow"