The Only Difference Between Text and Multimodal LLM Calls Is an Array
The jump from text to multimodal looks intimidating but is a one-line code change: swap a string for an array. Understanding that base64 is just JSON-compatible transport, not model comprehension, prevents a common category of confusion that leads developers to overcomplicate their pipelines.
Multimodal model calls use the same API endpoint and the same model as text calls. The sole difference is that `content` becomes an array of objects keyed by `type`—text, image_url, input_audio, or video_url—with binary data encoded as base64 strings. That base64 step is purely a transport mechanism to fit binary into JSON; the model's actual understanding happens later when a Vision Encoder splits images into patches and converts them to embeddings.
TTS works in the opposite direction. The API accepts text and returns a raw binary audio stream, not JSON. Saving that stream to an `.mp3` file and handing it to the system player is the entire client-side pipeline. Internally, a text encoder produces acoustic features like a mel spectrogram, and a vocoder such as HiFi-GAN turns those into waveform audio.
GPT-4o and similar models do not natively generate pixels or waveforms. When a user asks for an image or speech, the model plans the request and dispatches it to a separate module—DALL-E for images, a TTS engine for audio—then returns the result alongside its text response.
Many developers conflate base64 transport with model comprehension, which leads to unnecessary abstraction layers that re-encode or pre-process media the API already handles.
The clean separation between a reasoning model and dedicated generation modules means multimodal output is an orchestration problem, not a single-model capability, even though product branding obscures this.
TTS's binary response design is a practical reminder that not every AI API fits the familiar JSON-shaped mental model; handling raw streams is a necessary skill.