Recipes
All recipes
TypeScriptIntermediate

Gemini Agora Agentic Video for Next.js

Talk to a video with Gemini Agentic Video Understanding and Agora voice AI.

Source on GitHubView raw recipe

Recipe prompt

Paste into Cursor, Claude Code, v0, or your coding agent
Use with your coding agent
You are implementing the "Gemini Agora Agentic Video for Next.js" recipe in this project.

Read the recipe markdown first:
https://raw.githubusercontent.com/AgoraIO-Community/Gemini-Agora-Agentic-Video-NextJS/main/docs/ai/RECIPE.md

Use the source repository for cross-reference:
https://github.com/AgoraIO-Community/Gemini-Agora-Agentic-Video-NextJS

Build this recipe into the user's app using the markdown as the implementation guide. Inspect related source files through the repository links when the recipe points to them. Ask before installing new dependencies.

Recipe

Rendered from the configured recipe markdown.

Raw

Agora + Gemini Agentic Video Voice Assistant Recipe

Problem

Hackathon teams often want an AI assistant that can understand a video and then discuss it naturally. Building that from scratch usually means solving unrelated infrastructure first: realtime audio, speech recognition, agent lifecycle, TTS playback, video upload, model prompting, and timestamp grounding.

This starter packages those foundations while leaving the product idea open.

Architecture

Upload / Record Video
        |
Gemini Files API
        |
Gemini 3.7 Flash Agentic Video Understanding
        |
Structured JSON: summary, moments, context
        |
Agora Conversational AI agent instructions
        |
Voice conversation over Agora RTC + RTM

The key split is intentional:

  • Agora provides realtime conversation and communication.
  • Gemini provides video understanding and reasoning.

This is not a live continuous Gemini video-streaming architecture. The starter analyzes uploaded or recorded clips, then lets the user talk about the resulting structured context.

Agora Responsibilities

Agora owns the realtime voice experience:

  • RTC channel connection
  • RTM transcript/state/metrics events
  • token generation
  • Conversational AI agent invitation
  • agent lifecycle and stop flow
  • microphone publishing
  • agent audio playback
  • managed MiniMax TTS

The existing voice pipeline remains:

Microphone -> Gemini STT -> Gemini LLM -> MiniMax TTS -> Agora RTC -> Browser

Gemini Responsibilities

Gemini owns analysis and reasoning:

  • Files API upload for reusable video input
  • Gemini 3.7 Flash Interactions API
  • processing=agentic video understanding
  • structured summary and timeline generation
  • Gemini STT and Gemini LLM in the existing Agora voice pipeline

Agentic Video Workflow

The browser sends a selected MP4/WebM clip to app/api/video/analyze/route.ts.

The route:

  1. validates MIME type and size
  2. uploads the file to Gemini
  3. polls for PROCESSING, ACTIVE, or FAILED
  4. stops polling after a bounded timeout
  5. asks Gemini 3.7 Flash to analyze the video with agentic processing
  6. parses and validates the returned JSON with Zod
  7. returns structured context to the browser

The reusable prompt lives in lib/gemini/video-analysis-prompt.ts.

Video -> Context -> Conversation

After analysis, the frontend passes the validated video context into /api/invite-agent when starting the Agora Conversational AI session.

The invite route appends instructions telling the LLM:

  • it is discussing a previously analyzed video
  • the provided video context is the primary source
  • timestamps should be mentioned when useful
  • unknown details should not be invented
  • responses should stay concise for voice

This keeps the Agora architecture intact while grounding the voice agent in Gemini's video analysis.

Timestamp Grounding

Important Moment cards call video.currentTime = timestampSeconds, making timeline navigation reliable for the MVP.

The live transcript also detects common timestamp shapes such as 00:14 and makes them clickable when the agent says them.

Extension Ideas

Teams can replace the generic prompt with domain-specific analysis:

  • sports coach: form, repetitions, fatigue, positioning
  • classroom assistant: concepts, examples, confusing transitions
  • repair assistant: visible defects, state changes, unsafe steps
  • interview coach: clarity, confidence, pacing, missed signals
  • gaming coach: decisions, mistakes, timing, turning points
  • cooking assistant: technique, sequence, texture and doneness cues
  • accessibility assistant: factual scene narration and key changes

Keep the starter focused. Add product-specific depth in the prompt and UI copy before adding databases, auth, storage, dashboards, or RAG infrastructure.