Desktop Pet
Build a playful macOS companion you can interact with and talk to.
Implementation guide
In this guide
Desktop Pet 🐋
A tiny floating macOS companion who loves rice, chases a foxtail, and talks with you.
| Hang out | Feed rice | Rice frenzy | Foxtail play | Sleep & wake |
|---|---|---|---|---|
![]() | ![]() | ![]() | ![]() | ![]() |
Play together
Drag her title bar to bring her closer, offer a bowl of rice with Feed Rice, or choose Rice Frenzy to watch her eat with both hands and kick her feet.
Try Foxtail to tease her with a draggable grass toy. Hold it nearby for half a second and she pounces. Let her catch it, or pull away mid-jump and watch her miss. These interactions work locally without API credentials.
Click Sleep to watch her get drowsy and curl up beside you. Click her or Wake up to wake her; feeding and playing also wake her immediately. After 5 minutes without interaction in the pet window, she sleeps automatically. Sleep is disabled during voice calls.
Voice chat
Click Start Chat to talk with her. The demo connects DeepSeek and Fish Audio through Agora for real-time voice conversation. You can mute your microphone or end the call from the controls below the interaction buttons. See Voice setup to configure your own keys.
Requirements and quick start
The floating desktop app targets macOS. The current version has been tested on Apple Silicon; Intel and other macOS versions need validation. For other platforms, start with the browser preview.
1. Prepare your tools
Install Node.js 22+, Python 3, and Apple Command Line Tools. Check your terminal before proceeding.
node --version
python3 --version
xcrun --find swiftcIf Command Line Tools are missing, run xcode-select --install and wait for the installer to finish. If Node.js or Python is missing, install it and reopen your terminal. Then install pnpm.
npm install -g pnpm@10.11.02. Clone and launch
Run these commands in the directory where you want to keep the project.
git clone https://github.com/SyunWong/desktop-pet.git
cd desktop-pet
./scripts/setup.sh
./scripts/run-macos.shSetup installs the locked dependencies and creates voice/.env.local if it is missing. The run script builds and opens Desktop Pet.app. Expect a small floating character with interaction buttons and Start Chat. Try Feed Rice, Rice Frenzy, and a foxtail catch/miss first; these work without API keys. Drag the title to move her and click × to quit.
The build script compiles the web UI into voice/.next-desktop/ before building the native app. The app starts a local Next.js production server on a random loopback port, so development-only Issues indicators do not appear in the desktop window. Keep the generated app beside voice/. Close the existing window before running the launch script again. Rebuild after changing source code, voice/.env.local, or your Node installation. The app bundle needs this source directory and the installed Node packages to run.
Voice setup
The author chose the providers below for this demo. Developers can choose other STT, LLM, and TTS providers supported by their Agora integration, along with their preferred languages, models, and voices.
Configure Agora with the CLI
After running setup, use the Agora CLI to select a project and write its credentials. This custom repository uses agora project env write with an explicit Next.js layout and file path. You do not need to copy Agora credentials from the Console.
Install the CLI if needed (macOS/Linux):
curl -fsSL https://dl.agora.io/cli/install.sh | shCheck the installation, then log in to the global region used by this demo's Area.US configuration:
agora version
agora doctor --json
agora login --region global
agora project list --refresh-cacheThe commands below were checked against CLI 0.2.9. If your CLI does not support project env write --template, update it using the official installation instructions before continuing.
Choose an existing project from the list. If you need a new one, create it through the CLI first:
agora project create desktop-pet --feature rtc --feature convoaiFrom the repository root, replace YOUR_PROJECT_ID_OR_NAME with your chosen project's ID or exact name, then run:
agora project use "YOUR_PROJECT_ID_OR_NAME"
agora project feature enable convoai "YOUR_PROJECT_ID_OR_NAME"
agora project env write voice/.env.local --template nextjs --project "YOUR_PROJECT_ID_OR_NAME"
agora project doctor --feature convoaiThe explicit --project ensures the env file uses your chosen project even if the repository already has a different local binding. The CLI writes NEXT_PUBLIC_AGORA_APP_ID and NEXT_AGORA_APP_CERTIFICATE to voice/.env.local and records the local project binding in .agora/project.json. Keep both files local. Use the same env-write command when changing projects or refreshing credentials. Do not add --overwrite: the normal update preserves other settings, including provider keys.
Optional provider keys
To reproduce the included DeepSeek and Fish Audio demo, open voice/.env.local and add only your provider keys from those accounts, one NAME=value per line. The Agora values are managed by the CLI.
| Variable | Purpose |
|---|---|
NEXT_PUBLIC_AGORA_APP_ID | Agora project App ID, written by the CLI |
NEXT_AGORA_APP_CERTIFICATE | Matching App Certificate, written by the CLI and used server-side |
DEEPSEEK_API_KEY | Optional, selects the demo's DeepSeek LLM |
FISH_AUDIO_API_KEY | Optional, selects Fish Audio and the English greeting/persona |
Ensure the configured providers are available to your account. Calls can incur provider charges. Set both optional keys to reproduce the English voice demo. With neither optional key, the code uses Agora-managed OpenAI and MiniMax with a Chinese greeting.
Demo choices and customization
| Stage | Author’s demo choice | What you can customize |
|---|---|---|
| Speech recognition (STT) | Deepgram nova-3, zh-CN | Provider, recognition model and input language |
| Conversation (LLM) | DeepSeek deepseek-flash, thinking disabled | Provider, model, prompt and generation settings |
| Speech synthesis (TTS) | Fish Audio s2-pro, voice reference 091cad5003ed4e90826a92e1fadde278 | Provider, speech model, voice and output language |
The PET_SYSTEM_PROMPT constant in voice/app/api/invite-agent/route.ts defines the pet’s personality; it is an ordinary application variable, not an SDK identifier. Configure these stages in the same file, using the .withStt(...), .withLlm(...), and .withTts(...) sections. To switch providers, update the corresponding SDK provider configuration and its server-side environment variables. Changing models or voices within the same provider may only require updating that provider’s settings. Check support and availability for your account and SDK version.
The included code also ties the English greeting/persona to the presence of a Fish Audio key. When choosing another voice provider or language, update the prompt and greeting in the same file to match.
After saving your configuration, run the credential-presence check from the repository root.
cd voice
pnpm doctor
cd ..Doctor checks the local setup and that the two Agora variables have values. A successful call verifies credentials and provider access. Rebuild with ./scripts/build-macos.sh and reopen the desktop app after changing .env.local, then click Start Chat, allow microphone access, and speak. Expect an audible greeting and a response to your voice. Try Mute, Unmute, and End Call, then start a second call to verify reconnection. The macOS shell forwards audio to AVAudioEngine for native playback, including headphones. Browser previews use HTML audio playback.
Browser preview
Run setup first, then run the following from the repository root. Browser preview uses Node.js and pnpm; the Swift build is specific to the macOS desktop window.
cd voice
pnpm dev --hostname 127.0.0.1Open http://127.0.0.1:3000. The native floating window and native audio bridge are macOS features.
Development
Whale.swift— floating window, drag handle, local server and native audio bridge.voice/components/Pet.tsx— eating animations and foxtail interaction.voice/app/globals.css— pet animation styles.voice/app/api/invite-agent/route.ts— persona, model, voice and turn detection.voice/components/ConversationComponent.tsx— RTC/RTM lifecycle and audio playback.scripts/— setup and local app build.
Run these checks from the repository root. They can run with the blank configuration template.
cd voice
pnpm typecheck
pnpm verify:api
node scripts/verify-rtm-lifecycle.mjs
pnpm build:desktopAfter source or configuration changes, close the app, run ./scripts/build-macos.sh from the repository root, and reopen it. This rebuilds both the web UI and native executable. Browser development with pnpm dev still supports hot reload and uses .next/, separate from the desktop build. See validation results for checks and limitations of this source release.
Reproduce with your coding assistant
Give your AI access to this repository and paste the following task.
Set up and run this Desktop Pet repository on my machine.
Read README.md, scripts/setup.sh, and scripts/run-macos.sh first.
Check my OS and prerequisites. On macOS, follow the desktop quick start;
on another OS, use the browser preview and report that verification scope.
Run the existing setup script. Follow the README CLI flow to bind an Agora
project and write its credentials to voice/.env.local using
agora project env write with --template nextjs. Keep credentials local.
Start with local rice and foxtail interactions. Confirm the UI loads and
check both catching and missing the grass if browser controls are available.
For voice chat, use my chosen STT/LLM/TTS configuration. Consult the Voice
setup section and invite-agent route before changing providers. Ask me to
enter missing third-party provider keys locally, and report any unavailable provider access.
After code changes, run the documented checks. Report which startup, UI,
and real-call checks passed and which still need manual verification.For provider changes, look up the selected provider in the installed SDK and its current integration documentation. The customization map and original quickstart provenance are in the recipe.
Troubleshooting
- Startup failed or build missing — check
desktop.log, run setup, and run./scripts/build-macos.sh. - Red Issues badge in a desktop window — that is the Next.js development overlay. Rebuild and reopen the current app, which uses production mode. Real connection failures still appear in the pet’s status; tap a voice notice to dismiss it, or end and retry a failed call.
- Failed chat startup — partial RTM connections are closed before retry; ending a call or unloading the page also releases them.
- No microphone — allow access in macOS Privacy & Security, then restart the call.
- No sound — check system output and volume, use Enable Sound if shown, and reconnect.
- Voice connection fails — check credentials, Agora feature enablement and provider access.
- Moving the app to another folder breaks startup — keep it beside
voice/. - Clone outside protected Desktop/Documents folders to reduce first-run file-access prompts.
Data and deployment
Voice chat sends microphone audio to Agora and its configured speech/LLM providers. Keys stay in the local server environment. Local animation needs no cloud connection. The sample API routes have no user authentication, so keep the server on loopback; a hosted multi-user version needs authentication and rate limits.
Roadmap
Planned directions for future versions, with scope and timing still open.
- More expressions and reactions — closed eyes, happy, surprised, and richer responses to interaction.
- More characters — additional pets and a simpler way to switch character artwork, personalities, and voices.
- Tasks through conversation — tell your companion what you want to get done, connect it to tools, and hear progress and results in the same conversation. For example, ask it to summarize a document or organize a set of notes, with confirmation for actions that need it.
Ideas and contributions are welcome through GitHub issues and pull requests. See Contributing.
License and attribution
Project code is MIT licensed. This project derives its voice integration from Agora's official Next.js quickstart. Character artwork comes from the credited upstream pet project. See THIRD_PARTY_NOTICES.md and the preserved licenses. This is a community project.
# Desktop Pet 🐋
A tiny floating macOS companion who loves rice, chases a foxtail, and talks with you.
| Hang out | Feed rice | Rice frenzy | Foxtail play | Sleep & wake |
| :---: | :---: | :---: | :---: | :---: |
|  |  |  |  |  |
[中文说明](https://github.com/SyunWong/desktop-pet/blob/main/README.zh-CN.md) · [Recipe](https://github.com/SyunWong/desktop-pet/blob/main/docs/RECIPE.md) · [Credits](https://github.com/SyunWong/desktop-pet/blob/main/THIRD_PARTY_NOTICES.md)
## Play together
Drag her title bar to bring her closer, offer a bowl of rice with **Feed Rice**, or choose **Rice Frenzy** to watch her eat with both hands and kick her feet.
Try **Foxtail** to tease her with a draggable grass toy. Hold it nearby for half a second and she pounces. Let her catch it, or pull away mid-jump and watch her miss. These interactions work locally without API credentials.
Click **Sleep** to watch her get drowsy and curl up beside you. Click her or **Wake up** to wake her; feeding and playing also wake her immediately. After 5 minutes without interaction in the pet window, she sleeps automatically. Sleep is disabled during voice calls.
## Voice chat
Click **Start Chat** to talk with her. The demo connects DeepSeek and Fish Audio through Agora for real-time voice conversation. You can mute your microphone or end the call from the controls below the interaction buttons. See [Voice setup](#voice-setup) to configure your own keys.
## Requirements and quick start
The floating desktop app targets macOS. The current version has been tested on Apple Silicon; Intel and other macOS versions need validation. For other platforms, start with the browser preview.
### 1. Prepare your tools
Install Node.js 22+, Python 3, and Apple Command Line Tools. Check your terminal before proceeding.
```bash
node --version
python3 --version
xcrun --find swiftc
```
If Command Line Tools are missing, run `xcode-select --install` and wait for the installer to finish. If Node.js or Python is missing, install it and reopen your terminal. Then install pnpm.
```bash
npm install -g pnpm@10.11.0
```
### 2. Clone and launch
Run these commands in the directory where you want to keep the project.
```bash
git clone https://github.com/SyunWong/desktop-pet.git
cd desktop-pet
./scripts/setup.sh
./scripts/run-macos.sh
```
Setup installs the locked dependencies and creates `voice/.env.local` if it is missing. The run script builds and opens `Desktop Pet.app`. Expect a small floating character with interaction buttons and **Start Chat**. Try Feed Rice, Rice Frenzy, and a foxtail catch/miss first; these work without API keys. Drag the title to move her and click × to quit.
The build script compiles the web UI into `voice/.next-desktop/` before building the native app. The app starts a local Next.js production server on a random loopback port, so development-only Issues indicators do not appear in the desktop window. Keep the generated app beside `voice/`. Close the existing window before running the launch script again. Rebuild after changing source code, `voice/.env.local`, or your Node installation. The app bundle needs this source directory and the installed Node packages to run.
## Voice setup
The author chose the providers below for this demo. Developers can choose other STT, LLM, and TTS providers supported by their Agora integration, along with their preferred languages, models, and voices.
### Configure Agora with the CLI
After running setup, use the [Agora CLI](https://github.com/AgoraIO/cli) to select a project and write its credentials. This custom repository uses `agora project env write` with an explicit Next.js layout and file path. You do not need to copy Agora credentials from the Console.
Install the CLI if needed (macOS/Linux):
```bash
curl -fsSL https://dl.agora.io/cli/install.sh | sh
```
Check the installation, then log in to the global region used by this demo's `Area.US` configuration:
```bash
agora version
agora doctor --json
agora login --region global
agora project list --refresh-cache
```
The commands below were checked against CLI 0.2.9. If your CLI does not support `project env write --template`, update it using the official installation instructions before continuing.
Choose an existing project from the list. If you need a new one, create it through the CLI first:
```bash
agora project create desktop-pet --feature rtc --feature convoai
```
From the **repository root**, replace `YOUR_PROJECT_ID_OR_NAME` with your chosen project's ID or exact name, then run:
```bash
agora project use "YOUR_PROJECT_ID_OR_NAME"
agora project feature enable convoai "YOUR_PROJECT_ID_OR_NAME"
agora project env write voice/.env.local --template nextjs --project "YOUR_PROJECT_ID_OR_NAME"
agora project doctor --feature convoai
```
The explicit `--project` ensures the env file uses your chosen project even if the repository already has a different local binding. The CLI writes `NEXT_PUBLIC_AGORA_APP_ID` and `NEXT_AGORA_APP_CERTIFICATE` to `voice/.env.local` and records the local project binding in `.agora/project.json`. Keep both files local. Use the same env-write command when changing projects or refreshing credentials. Do not add `--overwrite`: the normal update preserves other settings, including provider keys.
### Optional provider keys
To reproduce the included DeepSeek and Fish Audio demo, open `voice/.env.local` and add only your provider keys from those accounts, one `NAME=value` per line. The Agora values are managed by the CLI.
| Variable | Purpose |
| --- | --- |
| `NEXT_PUBLIC_AGORA_APP_ID` | Agora project App ID, written by the CLI |
| `NEXT_AGORA_APP_CERTIFICATE` | Matching App Certificate, written by the CLI and used server-side |
| `DEEPSEEK_API_KEY` | Optional, selects the demo's DeepSeek LLM |
| `FISH_AUDIO_API_KEY` | Optional, selects Fish Audio and the English greeting/persona |
Ensure the configured providers are available to your account. Calls can incur provider charges. Set both optional keys to reproduce the English voice demo. With neither optional key, the code uses Agora-managed OpenAI and MiniMax with a Chinese greeting.
### Demo choices and customization
| Stage | Author’s demo choice | What you can customize |
| --- | --- | --- |
| Speech recognition (STT) | Deepgram `nova-3`, `zh-CN` | Provider, recognition model and input language |
| Conversation (LLM) | DeepSeek `deepseek-flash`, thinking disabled | Provider, model, prompt and generation settings |
| Speech synthesis (TTS) | Fish Audio `s2-pro`, voice reference `091cad5003ed4e90826a92e1fadde278` | Provider, speech model, voice and output language |
The `PET_SYSTEM_PROMPT` constant in `voice/app/api/invite-agent/route.ts` defines the pet’s personality; it is an ordinary application variable, not an SDK identifier. Configure these stages in the same file, using the `.withStt(...)`, `.withLlm(...)`, and `.withTts(...)` sections. To switch providers, update the corresponding SDK provider configuration and its server-side environment variables. Changing models or voices within the same provider may only require updating that provider’s settings. Check support and availability for your account and SDK version.
The included code also ties the English greeting/persona to the presence of a Fish Audio key. When choosing another voice provider or language, update the prompt and greeting in the same file to match.
After saving your configuration, run the credential-presence check from the repository root.
```bash
cd voice
pnpm doctor
cd ..
```
Doctor checks the local setup and that the two Agora variables have values. A successful call verifies credentials and provider access. Rebuild with `./scripts/build-macos.sh` and reopen the desktop app after changing `.env.local`, then click **Start Chat**, allow microphone access, and speak. Expect an audible greeting and a response to your voice. Try **Mute**, **Unmute**, and **End Call**, then start a second call to verify reconnection. The macOS shell forwards audio to AVAudioEngine for native playback, including headphones. Browser previews use HTML audio playback.
## Browser preview
Run setup first, then run the following from the repository root. Browser preview uses Node.js and pnpm; the Swift build is specific to the macOS desktop window.
```bash
cd voice
pnpm dev --hostname 127.0.0.1
```
Open http://127.0.0.1:3000. The native floating window and native audio bridge are macOS features.
## Development
- `Whale.swift` — floating window, drag handle, local server and native audio bridge.
- `voice/components/Pet.tsx` — eating animations and foxtail interaction.
- `voice/app/globals.css` — pet animation styles.
- `voice/app/api/invite-agent/route.ts` — persona, model, voice and turn detection.
- `voice/components/ConversationComponent.tsx` — RTC/RTM lifecycle and audio playback.
- `scripts/` — setup and local app build.
Run these checks from the repository root. They can run with the blank configuration template.
```bash
cd voice
pnpm typecheck
pnpm verify:api
node scripts/verify-rtm-lifecycle.mjs
pnpm build:desktop
```
After source or configuration changes, close the app, run `./scripts/build-macos.sh` from the repository root, and reopen it. This rebuilds both the web UI and native executable. Browser development with `pnpm dev` still supports hot reload and uses `.next/`, separate from the desktop build. See [validation results](https://github.com/SyunWong/desktop-pet/blob/main/docs/VALIDATION.md) for checks and limitations of this source release.
## Reproduce with your coding assistant
Give your AI access to this repository and paste the following task.
```text
Set up and run this Desktop Pet repository on my machine.
Read README.md, scripts/setup.sh, and scripts/run-macos.sh first.
Check my OS and prerequisites. On macOS, follow the desktop quick start;
on another OS, use the browser preview and report that verification scope.
Run the existing setup script. Follow the README CLI flow to bind an Agora
project and write its credentials to voice/.env.local using
agora project env write with --template nextjs. Keep credentials local.
Start with local rice and foxtail interactions. Confirm the UI loads and
check both catching and missing the grass if browser controls are available.
For voice chat, use my chosen STT/LLM/TTS configuration. Consult the Voice
setup section and invite-agent route before changing providers. Ask me to
enter missing third-party provider keys locally, and report any unavailable provider access.
After code changes, run the documented checks. Report which startup, UI,
and real-call checks passed and which still need manual verification.
```
For provider changes, look up the selected provider in the installed SDK and its current integration documentation. The customization map and original quickstart provenance are in [the recipe](https://github.com/SyunWong/desktop-pet/blob/main/docs/RECIPE.md).
## Troubleshooting
- Startup failed or build missing — check `desktop.log`, run setup, and run `./scripts/build-macos.sh`.
- Red Issues badge in a desktop window — that is the Next.js development overlay. Rebuild and reopen the current app, which uses production mode. Real connection failures still appear in the pet’s status; tap a voice notice to dismiss it, or end and retry a failed call.
- Failed chat startup — partial RTM connections are closed before retry; ending a call or unloading the page also releases them.
- No microphone — allow access in macOS Privacy & Security, then restart the call.
- No sound — check system output and volume, use **Enable Sound** if shown, and reconnect.
- Voice connection fails — check credentials, Agora feature enablement and provider access.
- Moving the app to another folder breaks startup — keep it beside `voice/`.
- Clone outside protected Desktop/Documents folders to reduce first-run file-access prompts.
## Data and deployment
Voice chat sends microphone audio to Agora and its configured speech/LLM providers. Keys stay in the local server environment. Local animation needs no cloud connection. The sample API routes have no user authentication, so keep the server on loopback; a hosted multi-user version needs authentication and rate limits.
## Roadmap
Planned directions for future versions, with scope and timing still open.
- **More expressions and reactions** — closed eyes, happy, surprised, and richer responses to interaction.
- **More characters** — additional pets and a simpler way to switch character artwork, personalities, and voices.
- **Tasks through conversation** — tell your companion what you want to get done, connect it to tools, and hear progress and results in the same conversation. For example, ask it to summarize a document or organize a set of notes, with confirmation for actions that need it.
Ideas and contributions are welcome through GitHub issues and pull requests. See [Contributing](https://github.com/SyunWong/desktop-pet/blob/main/CONTRIBUTING.md).
## License and attribution
Project code is MIT licensed. This project derives its voice integration from Agora's official Next.js quickstart. Character artwork comes from the credited upstream pet project. See `THIRD_PARTY_NOTICES.md` and the preserved licenses. This is a community project.




