AI Desktop Pet Creation: From a Static Image to a Dynamic Desktop Pet

It started simply: a friend showed me his Codex desktop pet—a little creature crouched in the corner of the screen, resting its chin on its hand when Codex was thinking, working when it ran tools, and giving a V-sign when the task was done. My immediate reaction was: "I want one too."

Why I Dared to Start

First, my background: I have some AI programming basics, but I'm definitely not a guru. Being able to make this desktop pet didn't rely on any profound skills, but rather on "breaking down steps + letting AI do the heavy lifting."

If you've also been tempted but felt you couldn't do it, this article is for you—because it really isn't that hard. I broke the whole process into four steps, each independently verifiable:

1. Find an image → Use AI to generate a dynamic video
2. Extract frames from the video → Turn them into a transparent-background "sprite sheet"
3. Use a desktop framework to "wrap" the sprite sheet → The desktop pet is born
4. Connect to Cursor / Claude Code "events" → It changes as you work

Let's go through them step by step.

Step 1: Make a Static Image "Move"

The core of a desktop pet is that "it must have actions." I don't have the ability to draw frame-by-frame by hand, so I took a shortcut: AI video generation.

I took my friend's image, threw it into an AI video generation tool, and had it generate action videos like the character "resting chin in thought," "raising hand in a V-sign," and "typing on a keyboard."

The tools I used were Doubao and Jimeng, chosen for their generous free quotas + natural movements + stable output.

There are a few real pitfalls in this step, let me clear them for you first:

• White background images: AI-generated videos often have a white background and may carry watermarks. White-background images can't be used directly as sprites; background removal is needed later.
• Camera drift: AI videos occasionally have a slow "zoom in / zoom out" effect, making the character suddenly larger or smaller. It's best to cut out unstable segments before extracting frames.
• Watermarks are semi-transparent: This is the most insidious point—on a white background, it's almost invisible, but after removing the background, it turns into a white blotch.

Step 2: Frame Extraction + Background Removal, Turning It into a "Sprite Sheet"

This step turns the video into material that a program can play. Let me explain a few terms in plain language:

• Frame extraction: A video is dozens of consecutive images per second (each called a frame). We don't need all of them; extracting one at intervals is enough for smooth motion.
• Background removal / matting: Remove the white background, leaving only the character with a transparent background (if possible, use the matting tools in PS or WPS, which work quite well).
• Sprite sheet: Stitch a long series of transparent images into one large image. The program "cuts" and plays them in sequence, creating animation.
• Loop segment: Find a segment of the action where the start and end can connect, allowing the desktop pet to loop "chin-resting" indefinitely, rather than stopping after one play.

For this step, I wrote a small script (Python, OpenCV for frame extraction + Pillow for background removal). The process is: extract frames → remove white background frame by frame → keep only the largest "connected region" (to prevent watermarks and noise from mixing in) → uniformly crop to the same bounding box → scale → automatically find loop segments → stitch into a sprite sheet.

Tip: Quantize the color palette (e.g., compress to 256 colors). For this kind of flat-shaded cartoon style, it's almost lossless but can reduce file size by about 85%. A 14MB image sequence compressed down to just 1.6MB.

At this point, you already have a bunch of playable "material," just one "shell" away from a desktop pet.

Step 3: Add a "Shell" to Let It Crouch in the Desktop Corner

For the shell, I used Tauri 2 + Svelte 5. Let me explain both terms:

• Tauri: A framework for making desktop applications (like .exe files on Windows). It's much lighter than the older Electron—my desktop pet uses only about 37MB of resident memory, while Electron often starts at 150MB+.
• Svelte: A UI framework, used with Canvas (the "drawing board" in HTML for graphics) to play sprite sheets and create animations like breathing, swaying, and bouncing.

This shell needs to do several things: a transparent, borderless, always-on-top window that doesn't appear in the taskbar; draggable and remembers its position; a right-click menu to switch skins; a system tray to show/hide/exit.

Tauri basically has ready-made capabilities for all of these. After AI helped me set it up, the desktop pet "crouched" in the bottom-right corner of my screen for the first time.

Step 4: Bring It "Alive"—Connecting Agent Events

A static desktop pet is boring. What truly makes it "mood-driven" is event-driven behavior.

The principle is this: Programming agents like Cursor / Claude Code / Codex all support something called hooks. You can think of them as "triggers"—at certain moments (starting to think, starting to execute a tool, task completion), the agent triggers this hook to call a command you specify.

My approach: Write a tiny bridge script and attach it to the hooks. Whenever the agent acts, the script POSTs a uniformly formatted message to the local 127.0.0.1:4271 port. The service inside the desktop pet receives it, forwards it to the frontend, and the frontend decides which animation to play and which speech bubble to display based on the state.

Thus, this linkage was born:

• Agent is thinking → Desktop pet rests chin on hand
• Agent is running a tool → Desktop pet spins a gear
• Agent is waiting for confirmation → Desktop pet waits
• Task completed → Desktop pet gives a V-sign and prompts "Click to dismiss"
• Error occurred → Desktop pet shows an exclamation mark and sweat drops

Here, I strictly adhered to one principle: The bridge script always exits quickly and never blocks the Agent. The desktop pet is just an observer; it must not jam up your work.

Current Status & Pitfalls I Encountered

Now the desktop pet is running stably, supporting Claude Code / Codex / Cursor. It can switch skins, scale, lock monitoring to a specific session, auto-start on boot, and auto-update.

A few pitfalls I stepped into:

1. Other desktop pets can "steal" your hooks: I had previously installed another desktop pet on my machine, which would overwrite ~/.claude/settings.json and knock off my attached hooks. The symptom was "the desktop pet suddenly stopped receiving events."
2. Transparent windows have shadow borders: On Windows, transparent windows need to have shadows turned off, otherwise there's a thin border around them, which looks ugly. Just one line of configuration.
3. Don't be fooled by "no response": For a while, I thought the completion state wasn't displaying. After troubleshooting for ages, it turned out the "completion" animation only played for a fraction of a second before returning to idle, invisible to the naked eye. The state hold time needed to be extended.

Where I'm Stuck, Seeking Expert Advice

Writing up to here, the desktop pet is already a qualified "mood display." But my desired next step is: Let it go from "watching" to "talking"—converse with it via voice or text, and let it help me control the computer.

The ideal form is: I say "Help me open the browser to search for something or open a program," it understands, executes, and gives me feedback through the desktop pet's expressions and bubbles.

Where am I stuck? On "how to make the desktop pet a true execution entry point"—should I embed a dialogue/voice loop directly inside the desktop pet, or reuse an existing Agent (letting it call Claude Code / Codex capabilities)? Which voice solution to use (local ASR or cloud interface)? How to handle the permission boundaries and security of desktop control?

On this front, I'm currently in a state of "having some ideas but unsure about the technical route." If you've worked on a similar "desktop pet + voice assistant / desktop control" project, or have recommended solutions, please point the way in the comments. I'd be immensely grateful.

Finally

My biggest takeaway from the whole process is: "Looks hard" and "is actually hard" are two different things. Once broken down, every step has ready-made tools and AI to carry the load for you. You just need to be the person who "strings the steps together."

Have you ever been tempted to make a desktop pet? Or made some small project that "others find useless, but you had a blast making"? Come chat in the comments—maybe your story will be our next article.