The Only Difference Between Text and Multimodal LLM Calls Is an Array

Multimodal Models and TTS: From Concept to Code, Explained Clearly

Foreword

When working on RAG projects, we typically encounter text vectorization (Embedding) and semantic retrieval. But AI's capabilities go far beyond text—images, audio, and video can all serve as model inputs and outputs. This article helps you understand two things at the code level: multimodal models and TTS (Text-to-Speech).

1. Multimodal Models vs TTS: Two Concepts on Different Dimensions

Dimension	Multimodal Model	TTS
Core Capability	Cross-modal understanding and generation (image→description, text→image, audio→text...)	Text→Speech (single direction)
Input	Text / Image / Audio / Video	Text only
Output	Text / Image / Audio	Audio only (mp3/wav etc.)
Representatives	GPT-4o, Gemini, Claude 3.5	OpenAI TTS, ElevenLabs, Edge TTS
Complexity	Combined understanding + generation capability	Single modal conversion

TTS is a small branch of multimodal capability—the text→speech direction. Multimodal models are a more general framework.

2. Code Implementation of Multimodal Models

2.1 Core: `content` Changes from a String to an Array

In pure text calls, content is just a string:

import { HumanMessage } from "@langchain/core/messages";

// Pure text
new HumanMessage({
    content: "What's in this picture?"
})

// Corresponding HTTP request body:
// {
//     "role": "user",
//     "content": "What's in this picture?"
// }

In multimodal calls, content becomes an array, using type to distinguish different media types:

import fs from "fs";

const imageBase64 = fs.readFileSync("./cat.jpg").toString("base64");

new HumanMessage({
    content: [
        { type: "text", text: "What's in this picture? Please describe it in Chinese." },
        {
            type: "image_url",
            image_url: {
                url: `data:image/jpeg;base64,${imageBase64}`
            }
        }
    ]
})

// Corresponding HTTP request body:
// {
//     "role": "user",
//     "content": [
//         { "type": "text", "text": "What's in this picture? Please describe it in Chinese." },
//         { "type": "image_url", "image_url": { "url": "data:image/jpeg;base64,..." } }
//     ]
// }

This is the only difference between pure text calls and multimodal calls. The model is still the same (e.g., gpt-4o), the API endpoint hasn't changed, only the content field has changed from string to ContentPart[].

2.2 The Unified `content` Array Pattern

All non-text media inputs follow the same set of rules—use type to distinguish types, use base64 to transmit binary data:

Input Type	content Array Element	Data Format
Text	`{ type: "text", text: "..." }`	Native string
Image	`{ type: "image_url", image_url: { url: "data:..." } }`	base64
Audio	`{ type: "input_audio", input_audio: { data: "...", format: "wav" } }`	base64
Video	`{ type: "video_url", video_url: { url: "data:..." } }`	base64

You can even send multiple images at once:

new HumanMessage({
    content: [
        { type: "text", text: "What's the difference between Image 1 and Image 2?" },
        { type: "image_url", image_url: { url: "data:image/png;base64,image1..." } },
        { type: "image_url", image_url: { url: "data:image/png;base64,image2..." } }
    ]
})

3. Why Convert Non-Text to base64?

Because HTTP APIs can only transmit text. OpenAI's API is a RESTful interface, the request body is JSON, and JSON cannot directly contain binary data.

Three ways to pass images/audio:

// Method 1: base64 encoding (most common, data embedded directly in the request)
{ image_url: "data:image/jpeg;base64,/9j/4AAQSkZJRg..." }

// Method 2: Public URL (the model downloads it itself)
{ image_url: "https://example.com/cat.jpg" }

// Method 3: Upload first to get a file_id, then reference it
{ image_id: "file-abc123" }

base64 essentially represents binary byte streams using ASCII characters, so they can be stuffed into JSON strings for transmission. The image content doesn't change at all—a 512×512 cat picture is still the same cat after base64 conversion.

4. base64 Transmission ≠ Model Understanding Images

These are two completely different stages, something many beginners easily confuse:

Stage 1: Client → API Server (base64 transmission)
Original image binary: FF D8 FF E0 00 10 4A 46 ...
      ↓ base64 encoding
JSON string:          "/9j/4AAQSkZJRgABAQ..."
      ↓ Server-side decode
Restored to binary:   FF D8 FF E0 00 10 4A 46 ...

Stage 2: Server-side model internals (truly "understanding" the image)
Original image (512×512×3 RGB)
      ↓ Split into patches (16×16 grid, each 32×32 pixels)
Total 256 patches
      ↓ Each patch flattened into a 1D vector
      ↓ Through Vision Encoder (ViT)
Each patch → visual embedding
      ↓ Concatenated with text token embeddings
      ↓ Fed into Transformer for cross-attention → output text

	base64 Encoding	Patch + Vision Encoder
Who does it	Your client code	OpenAI/Cloud server
Purpose	Allows images to be transmitted via JSON	Allows the model to "understand" the image
Image changes	Format changes, content unchanged	Broken down into numerical vectors
Stage	Before sending the request	During model inference

Simple analogy: base64 is the courier putting a package into a cardboard box; patch splitting is the recipient opening the box and reading the contents. The two things are completely unrelated.

5. TTS Code Implementation

5.1 Basic Usage

import OpenAI from "openai";
import fs from "fs";
import { exec } from "child_process";

const openai = new OpenAI({
    apiKey: process.env.OPENAI_API_KEY,
    baseURL: process.env.OPENAI_BASE_URL,
});

// Call TTS API
const mp3 = await openai.audio.speech.create({
    model: "tts-1",           // or "tts-1-hd" for high quality
    voice: "shimmer",         // timbre: alloy / echo / fable / onyx / nova / shimmer
    input: "Hello, the weather is really nice today, perfect for a walk.",
    speed: 1.0,               // speed 0.25 ~ 4.0
});

// Write the audio binary stream to a file
const buffer = Buffer.from(await mp3.arrayBuffer());
fs.writeFileSync("./output.mp3", buffer);

// Play audio (Windows)
exec("start output.mp3");
console.log("Speech generation complete, now playing...");

5.2 Line-by-Line Analysis

// mp3.arrayBuffer() → Reads raw binary data from the API response (the 0101 byte stream of audio)
// Buffer.from() → Wraps the binary data into a Node.js Buffer object
const buffer = Buffer.from(await mp3.arrayBuffer());

// Writes the Buffer to a local .mp3 file (sync means it waits until writing is done before proceeding)
fs.writeFileSync("./output.mp3", buffer);

// Windows uses the default program to open .mp3 (calls the system music player to play)
exec("start output.mp3");

One-sentence summary: Get raw audio bytes from the API → save as an mp3 file → let the system open it with the default player.

5.3 The TTS API Does Not Return JSON

This is the biggest difference from the Chat API:

// Chat API — returns JSON, you can directly access properties
const chatResponse = await openai.chat.completions.create({
    model: "gpt-4o",
    messages: [{ role: "user", content: "Hello" }]
});
console.log(chatResponse.choices[0].message.content);  // "Hello! How can I help you?"

// TTS API — returns a binary stream (essentially a Fetch Response object)
const audioResponse = await openai.audio.speech.create({
    model: "tts-1",
    voice: "shimmer",
    input: "Hello"
});
// audioResponse.content is undefined!
// Only the body property exists, containing the raw mp3 byte stream

Why this design? Audio files are large; if encoded into JSON (like base64), the size would inflate by 33%, and JSON parsing itself has overhead. Returning the raw binary stream directly is the most efficient.

5.4 TTS Internal Principles (Simplified)

Text "Hello, the weather is really nice today"
      ↓ Text Encoder (similar to BERT)
Acoustic Features (Mel Spectrogram)
      ↓ Vocoder (e.g., HiFi-GAN)
Waveform Audio (mp3/wav byte stream)
      ↓ Returned to client

6. How Does the "Output" of Multimodal Models Work?

Many people think multimodal models can directly output images and audio, but the actual current implementation works like this:

You send text "Read this passage in a gentle voice"
      ↓
GPT-4o first generates a text response
      ↓
Internally calls the TTS module (audio synthesis)
      ↓
Returns: text + audio

You send an image "Draw a comic version of this cat"
      ↓
GPT-4o first understands the image content
      ↓
Generates a drawing prompt → internally calls DALL-E
      ↓
Returns: description text + generated image

Key point: GPT-4o itself does not perform pixel-level image generation or waveform-level audio synthesis; it does "understanding + planning," then dispatches specialized generation modules. OpenAI packages them under the same brand, making it seem to users like one model can output anything.

Output Type	API Method
Text	Directly returns `response.content`
Image	API returns image URL or base64
Audio	Separate call to TTS API (`openai.audio.speech.create`)

So at the code level, text output is unified, while image and audio output typically go through separate APIs.

7. Complete Architecture Summary

Stringing the entire process together:

┌─────────────────────────────────────────────────────┐
│                    Your Client Code                   │
├─────────────────────────────────────────────────────┤
│                                                       │
│   ┌──────┐   base64 encode   ┌──────────┐            │
│   │ Image │ ───────────────→  │ JSON Req  │ ──── HTTP ──→│
│   └──────┘                   └──────────┘            │
│   ┌──────┐   base64 encode   ┌──────────┐            │
│   │ Audio │ ───────────────→  │ JSON Req  │ ──── HTTP ──→│
│   └──────┘                   └──────────┘            │
│                                                       │
│   ←── Text JSON ────  Chat API Response               │
│   ←── Image URL ────  Generation Result               │
│   ←── Audio Binary ───  TTS API Response → Save File → Play │
└─────────────────────────────────────────────────────┘
                           │
                           ↓
┌─────────────────────────────────────────────────────┐
│                   API Server Side                     │
├─────────────────────────────────────────────────────┤
│   base64 decode → Vision/Audio Encoder → Vector      │
│   → Transformer Cross-Attention → Text Output        │
│   → Or dispatch DALL-E / TTS and other specialized modules │
└─────────────────────────────────────────────────────┘

8. Remember in One Sentence

Multimodal: content changes from string to array, using { type, ... } to distinguish media types, base64 for transmission, encoder for understanding. The calling method is almost unchanged.
TTS: Input text, API returns an audio stream, directly save to file and play. Unlike the Chat API, the response is binary, not JSON.

I hope this sufficiently long article gives you a clear understanding of multimodal and TTS. If it was helpful, likes and bookmarks are welcome.