An AI Podcast Agent Gains Video Recognition and a Self-Cleaning Storage Layer

Project Introduction

This is an AI-driven podcast creation assistant focused on the full workflow from text/video/audio content recognition to podcast production. You can converse with it using natural language, or upload videos and audio; it will understand your intent, invoke the corresponding speech synthesis tools, and turn them into real podcast audio.

For a detailed introduction to the project, see the following articles:

First AI Agent Project: Building an AI Audio Creation Assistant from Scratch (Beginner's Guide): This is the first version, which mainly covers the initial framework setup, including a Vue3 + ai-elements-vue frontend interface, and a backend driven by LangChain 1.0 × AG-UI Protocol × Qwen TTS, supporting custom timbres, voice cloning, and dubbing functions.
First AI Agent Project: Building an AI Audio Creation Assistant from Scratch (Advanced Guide): This is the second version, which added complete podcast post-production capabilities, including three core functions: audio splicing, intelligent BGM selection, and track mixing. It also added an audio resource management API and voice input functionality, achieving a full-chain closed loop from frontend upload and voice input to backend processing.

This article focuses more on the features added in the high-level guide.

Update Overview

This update adds audio and video recognition capabilities, a temporary-permanent dual-layer storage architecture, a scheduled cleanup mechanism, and a smarter prompt system, upgrading the podcast Agent from a simple "speech synthesis tool" to a "full-chain audio content creation platform."

1. Backend Updates

1. Audio/Video File Storage Path Restructuring

Changed File: backend/app/tools/audio_index.py

Previously, audio files were uniformly saved in the storage/audios/ directory. Now they are saved to the temporary directory storage/temp/, achieving separate management of temporary and permanent files.

Core Change: Added the save_media_to_temp() function, supporting three input types:

def save_media_to_temp(media_data, filename: str) -> str:
    """Save audio/video files to the temporary directory"""
    temp_dir = STORAGE_DIR / "temp"
    temp_dir.mkdir(parents=True, exist_ok=True)
    file_path = temp_dir / filename

    if isinstance(media_data, str):
        # base64 string: decode and save
        file_data = base64.b64decode(media_data)
        file_path.write_bytes(file_data)
    elif isinstance(media_data, bytes):
        # bytes object: save directly
        file_path.write_bytes(media_data)
    else:
        # AudioSegment object: export file
        file_format = filename.split('.')[-1]
        media_data.export(str(file_path), format=file_format)

    return f"storage/temp/{filename}"

Design Rationale: Intermediate products generated by tools like timbre design and voice cloning are uniformly placed in storage/temp/. Once the user confirms satisfaction, the timbre is permanently migrated to storage/audios/ via the save_voice tool. This avoids permanent directory bloat and makes file lifecycle management clearer.

2. Scheduled Temporary File Cleanup Task

Changed File: backend/app/utils/temp_cleanup.py (New)

A new scheduled cleanup mechanism automatically removes expired files from the storage/temp/ directory that exceed a specified time limit, preventing temporary file accumulation from consuming disk space.

Core Implementation:

def cleanup_temp_files(max_age_minutes: int = 10) -> dict:
    """Clean up expired temporary files"""
    cutoff_time = time.time() - (max_age_minutes * 60)

    for file_path in TEMP_DIR.rglob("*"):
        if not file_path.is_file():
            continue
        file_mtime = file_path.stat().st_mtime
        if file_mtime < cutoff_time:
            file_size = file_path.stat().st_size
            file_path.unlink()
            stats["deleted_files"] += 1
            stats["freed_bytes"] += file_size

    # Clean up empty directories
    cleanup_empty_dirs(TEMP_DIR)
    return stats

Key Features:

Default retention time of 10 minutes, flexibly configurable
Recursively cleans all subdirectories and automatically removes empty directories
Returns detailed statistics (file count, deletion count, freed space)
Provides a schedule_cleanup_task() function for easy integration with scheduling frameworks like APScheduler

3. Voice Saving Tool

Changed File: backend/app/tools/voice_save.py (New)

A new save_voice tool permanently saves user-approved custom timbres from the temporary directory to storage/audios/ and records them in the voice_index.json index file.

Core Logic:

@tool("save_voice", args_schema=VoiceSaveInput)
def save_voice_tool(audio_source: str, voice_id: str, text: str, model_name: str = "") -> str:
    # Determine input type: file path or base64 data
    if audio_source.startswith("storage/") or audio_source.startswith("/"):
        # Copy from temporary directory to permanent directory
        source_path = BASE_DIR / audio_source.lstrip("/")
        dest_path = AUDIOS_DIR / source_path.name
        shutil.copy2(source_path, dest_path)
        local_path = f"storage/audios/{source_path.name}"
    else:
        # base64 data: decode and save
        local_path = save_audio_from_base64(audio_source, text, "saved_voice")

    # Record to index file
    record_voice_index(local_path, voice_id, model_name, path="audios")
    return json.dumps({"audio_url": local_path, "voice_id": voice_id, ...})

Design Highlights:

Supports two input methods: temporary file path (direct copy) and base64 encoded data (decode and save)
Automatically generates unique filenames containing timestamps, UUIDs, voice IDs, and text snippets
Uses file locks (fcntl.flock) to ensure concurrent write safety for the index file
Seamlessly integrates with the podcast workflow: timbre design → temporary save → user confirmation → permanent save

4. Speech Recognition Tool (ASR)

Changed File: backend/app/tools/qwen_asr.py (New)

Integrates the Alibaba Cloud DashScope qwen3-asr-flash model, providing high-precision speech recognition capabilities.

Core Call Chain:

@tool("qwen_asr_tool", args_schema=ASRInput)
def qwen_asr_tool(audio: str, enable_itn: bool = False) -> str:
    # 1. Parse audio source (URL / local path → data URI)
    audio_url = _resolve_audio_source(audio)

    # 2. Call ASR API
    response = _call_asr_api(audio_url, enable_itn)

    # 3. Extract recognized text
    text = _extract_text_from_response(response)
    return text

Key Capabilities:

Multi-format support: MP3, WAV, OGG, FLAC, M4A, AAC, and other mainstream audio formats
ITN (Inverse Text Normalization): Automatically converts spoken expressions to written form (e.g., "二零二五年" → "2025年")
Intelligent source resolution: Supports local paths and network URLs; local files are automatically converted to base64 data URIs
Automatic MIME type inference: Matches the correct MIME type based on file extension

5. Multimodal Recognition Tool

Changed File: backend/app/tools/qwen_multimodal.py (New)

Integrates the Alibaba Cloud DashScope qwen3.5-omni-plus model, supporting unified multimodal understanding of images, videos, and audio.

Two Core Tools:

Tool Name	Function	Applicable Scenario
`qwen_multimodal_tool`	Single-media multimodal recognition	Analyze a single image/video/audio clip
`qwen_combined_multimodal_tool`	Combined multimodal recognition	Analyze multiple media simultaneously (e.g., image + audio combination)

Large Video Intelligent Segmentation: When a video file exceeds 21MB, it is automatically split into multiple segments using moviepy for separate processing:

def _split_and_encode_video(video_path: Path, suffix: str) -> List[dict]:
    video = VideoFileClip(str(video_path))
    target_size = 10 * 1024 * 1024  # 10MB per segment
    num_segments = max(1, int(file_size / target_size))
    segment_duration = duration / num_segments

    for i in range(num_segments):
        segment = video.subclipped(start_time, end_time)
        # Encode and add to segment list
        segments.append(_build_media_content(data_uri, suffix, is_url=False))
    return segments

Additional Capabilities:

Supports streaming output (stream=True) for real-time recognition results
Supports audio output modality (enable_audio_output), allowing voice synthesis with specified timbres
Comprehensive error handling and log tracing

6. Prompt Optimization

Changed File: backend/app/services/prompt.py

The system prompt for the podcast Agent has been comprehensively refined and enhanced:

Optimization Highlights:

Structural Reorganization: The previously cluttered prompt was split into clear modular structures—core capability positioning, working methods, tool invocation rules, communication norms, context understanding, execution flow
New Audio/Video Recognition Content: Clearly distinguishes the usage scenarios for speech recognition (qwen_asr_tool) and multimodal understanding (qwen_multimodal_tool / qwen_combined_multimodal_tool) within the workflow
Tabular Tool Usage Strategy: Uses clear mapping relationships to explain each tool's responsibilities, reducing the probability of LLM misinvocation
Standardized Voice Saving Process: Clearly defines the four-step process of "Design → Temporary Save → User Confirmation → Permanent Save"
Avoiding Duplicate Calls: Emphasizes in the prompt that "for simple requests, end the reply immediately after calling the tool once"

New Audio/Video-to-Podcast Workflow Description:

**Audio/Video to Podcast Workflow**:
1. Content Recognition: Call qwen_multimodal_tool or qwen_asr_tool to recognize content in the audio/video
2. Script Arrangement: Based on the recognized content, organize it into a podcast script
3. Podcast Production: Continue executing according to the workflow steps to complete podcast production

2. Frontend Updates

There are no frontend changes in this update, focusing mainly on the expansion and optimization of backend capabilities. The existing ChatAgent.vue on the frontend already possesses capabilities for audio/video file upload, playback preview, and tool call result display, fully capable of supporting the newly added ASR and multimodal recognition features on the backend.

3. Architecture Summary

The core thread of this update:

User uploads audio/video
        │
        ▼
┌─────────────────────────────────────────────┐
│  Content Recognition Layer                  │
│  ├─ qwen_asr_tool         Speech→Text       │
│  └─ qwen_multimodal_tool  Audio/Video→Understanding │
└─────────────────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────────────────┐
│  Podcast Production Layer                   │
│  ├─ Timbre Design → Temp Dir (storage/temp/)│
│  ├─ save_voice → Permanent Dir (storage/audios/) │
│  └─ Audio Synthesis / Mixing / Splicing     │
└─────────────────────────────────────────────┘
        │
        ▼
┌─────────────────────────────────────────────┐
│  System Maintenance Layer                   │
│  └─ temp_cleanup  Scheduled cleanup of temp files │
└─────────────────────────────────────────────┘

Through the "temporary-permanent" dual-layer storage architecture + scheduled cleanup mechanism, both the flexibility of the podcast production process and effective management of system resources are ensured.

4. Technical Highlights

1. Temporary-Permanent Dual-Layer Storage Architecture

The core architectural design of this update. Intermediate products generated by tools (timbre designs, voice cloning results) are uniformly written to storage/temp/. Only timbres explicitly confirmed as satisfactory by the user are migrated to storage/audios/ via the save_voice tool. This design:

Avoids Storage Bloat: Temporary files have a 10-minute lifecycle and are automatically cleaned up upon expiration
Clear Semantics: temp/ = unconfirmed intermediate products, audios/ = user-confirmed final products
User-Controllable: The user has the initiative to "save" or "discard"

2. Large Video Intelligent Segmentation Strategy

In qwen_multimodal.py, when a video file exceeds 21MB, it is automatically split evenly along the timeline into multiple ≤10MB segments using moviepy for separate upload and recognition. This solves the limitation of large model APIs on single file size while ensuring recognition completeness.

3. Concurrency-Safe Index File Writing

voice_save.py uses fcntl.flock to apply an exclusive lock on voice_index.json, ensuring no data races or index corruption occur during concurrent writes from multiple requests.

4. Modular Prompt Engineering

prompt.py splits the system prompt into 6 independent modules (capability positioning, working methods, tool rules, communication norms, context understanding, execution flow). Each module has a single responsibility and clear boundaries, significantly reducing the probability of LLM misinvoking tools and facilitating subsequent iterative maintenance.

5. Results Display

Comments

Top 1 from juejin.cn, machine-translated. The original thread is authoritative.

breeze微风

Is it open source? I want to try it out.