AI Image Generation Is More Than Just Calling an API: The Design and Engineering Trade-offs of a Production-Grade Image Generation Pipeline

Author: Qingyan Tags: Agent, Next.js

This article is based on the actual implementation of the AI Mind project. GitHub: https://github.com/HWYD/ai-mind Corresponding code version: v0.4.12 Online experience: https://ai.hwyblog.cloud/instant-mind

AI Mind is a continuously iterating Next.js AI Chat project. Starting from basic local chat, it gradually added streaming protocols, tool calls, MCP, Skill, and Agent capabilities.

If you are interested in this project, or if this article has helped you a little, you are also welcome to give AI Mind a Star⭐ on GitHub. This will be a great encouragement for me to continue updating.

---

1. Behind a Single Image Generation Fetch, Three Real Problems Hide

A user types /image an orange cat sitting on a windowsill, sunlight coming in from the left. A few seconds later, an image appears on the screen.

It looks simple: send the text to an image generation API, display the returned image. Just one fetch call.

But when you actually start building it, three problems emerge one after another, each harder to gloss over than the last:

1. User description ≠ executable image generation prompt. "An orange cat sitting on a windowsill" — what is the scene? What is the style? Is the aspect ratio horizontal or square? The image generation model needs this information, but the user didn't provide it. There must be a step that completes the natural language into structured requirements, without disguising system defaults as user demands.
2. The prompt generated by the LLM may deviate from the user's intent. When you ask an LLM to rewrite an ImageBrief into an image generation prompt, it might miss the key constraint "sunlight coming in from the left," or it might take the liberty of changing "orange cat" to "orange tabby cat." A step is needed to check the prompt against the original requirements, and the check itself must be verifiable — it cannot rely on the LLM simply saying "I think it's fine."
3. The temporary URL returned by the Provider is untrustworthy. The image generation API returns an HTTPS URL saying "the image is here." Where does this URL point? Will it redirect? Is the content really an image? Could the file size be too large? Can you dare to give this URL directly to the frontend?

These three problems, plus the invisible defense line that "the pipeline itself cannot spiral out of control," together form four boundary points where hard decisions must be made. This article uses the implementation of AI Mind v0.4.12 as a thread to unfold them one by one — the focus is not on "what was done," but on "why it was done this way, and what problems would arise if it weren't."

---

2. Pipeline Overview — Three-Layer Architecture and Separation of Concerns

The image generation pipeline in v0.4.12 is divided into three layers, each with its own hard boundaries.

Layer 1: ChatOrchestrator (Main Chat Dispatcher)

ChatOrchestrator is the entry orchestrator for the AI Mind chat system. Its responsibility is to identify the /image command and divert the request before it enters the normal chat pipeline. The key judgment: only messages where the first non-whitespace token is exactly /image enter the image generation flow. /imagex, embedded /image, or "draw me a picture" in normal chat do not trigger it and continue to the normal chat path.

Layer 2: ImageGenerationRunCoordinator (Image Generation Task Lifecycle Manager)

The Coordinator is the actual executor of the image generation pipeline. It holds all side-effect dependencies — database, network, stream writer — and is responsible for orchestrating Graph execution, calling the Provider, managing streaming output, and error handling. Domain decisions are delegated to the Graph, while side-effect operations (persistence, network calls, streaming output) are completed by itself.

Layer 3: LangGraph StateGraph (Pure Domain Decision Graph)

The Graph only holds serializable domain state and does not touch the database, network, or streaming output. Its responsibilities are: extracting a structured ImageBrief from the user description, generating an execution prompt, checking prompt quality, deciding whether revision is needed, and deciding whether to allow generation.

The data flow between the three layers is unidirectional:

ChatOrchestrator → identifies /image → creates StreamRun
  → ImageGenerationRunCoordinator → orchestrates Graph execution
    → StateGraph → pure decisions (brief / draft / inspect / revise / block)
    ← returns GraphState (domain decision results)
  → Coordinator → calls Provider / streams output / persists
→ Frontend → streaming consumption / preview / download

With this layering, the Graph can be tested independently — using a mock planning model and image provider, all branches (pass / revise / block / counter limit) can be verified without starting a database or network. At the same time, atomic operations in the Coordinator (like changing generationCount from 0 to 1 and then calling the Provider) will not be re-executed due to retries of Graph nodes — the Graph holds no side-effect references, so node retries will not trigger duplicate network calls.

Image placeholder: Three-layer architecture relationship diagram — the responsibility boundaries and data flow of ChatOrchestrator / ImageGenerationRunCoordinator / StateGraph.

---

3. Structured Extraction — Using Zod Schema to Make LLM Output Verifiable Data

The first boundary point: How does the user's natural language input become structured data usable by subsequent steps?

Why ImageBrief is Needed

The user says "an orange cat sitting on a windowsill, sunlight coming in from the left." What information is in this sentence?

• Subject: orange cat (explicitly stated by the user)
• Action/Position: sitting on a windowsill (explicitly stated by the user)
• Lighting: sunlight coming in from the left (explicitly stated by the user)
• Scene: indoor windowsill (inferable, but not explicitly stated by the user)
• Style: unspecified
• Composition: unspecified
• Aspect Ratio: unspecified

If you directly ask an LLM to rewrite this sentence into an image generation prompt, the LLM will fill in the "unspecified" values with its own guesses — and it might guess differently each time. More dangerously, it might miss the key constraint "sunlight coming in from the left."

Therefore, an intermediate product is needed: ImageBrief — a structured factual record that distinguishes "what the user explicitly requested" from "reasonable system defaults." It serves as the sole benchmark for subsequent prompt generation and quality checks.

Schema Design: Boundary Numbers and the strict Strategy

// Problem solved: Constrain LLM output boundaries to prevent uncontrolled generation; distinguish "user requirements" from "system defaults"
export const imageBriefSchema = z
  .object({
    aspectRatio: z.enum(['square', 'landscape', 'portrait']),
    intent: z.string().trim().min(1).max(160),
    subjects: z.array(z.string().max(120)).min(1).max(8),
    mustInclude: z.array(z.string().max(160)).max(12),
    avoid: z.array(z.string().max(160)).max(12),
    assumptions: z.array(z.string().max(160)).max(8),
    scene: z.string().max(240).optional(),
    composition: z.string().max(240).optional(),
    style: z.string().max(240).optional(),
    lightingAndColor: z.string().max(240).optional(),
    visibleText: z.array(z.string().max(120)).max(8).optional(),
  })
  .strict()

Each boundary number has a specific constraint behind it: intent ≤ 160 characters (the image generation intent should be one sentence), subjects ≤ 8 items (a single image cannot have more than 8 identifiable subjects), mustInclude ≤ 12 items (if a user has more than 12 "must-satisfy" constraints, the requirement itself should be split).

.strict() is an easily overlooked detail. It rejects any fields in the LLM output that are not in the schema. If the LLM hallucinates a mood or cameraAngle field, it will not silently pass through but will trigger a parsing failure. After a parsing failure, the system does not perform hidden JSON repair or retries; it directly terminates with IMAGE_PROMPT_PLANNING_FAILED.

assumptions: Distinguishing "User Requirements" from "System Defaults"

The role of the assumptions field is to record the default assumptions the system made on behalf of the user and mark them as "this is not what the user requested."

The user did not specify an aspect ratio, so the system defaults to square. This default value is written into assumptions: ["Default square aspect ratio"]. When the subsequent prompt inspection step sees this field, it knows that "the aspect ratio is a system default, not a user requirement," and will not block generation due to an aspect ratio issue.

If assumptions did not exist, system defaults and user requirements would be mixed together, and the inspection step would not be able to distinguish "was this requirement unmet because the user didn't mention it or because the LLM missed it?" This field bears the responsibility of "factual attribution" throughout the entire pipeline.

---

4. Prompt Quality Inspection — Replacing "I Think It's Fine" with Structured Judgment

The second boundary point: How to confirm that the image generation prompt generated by the LLM is faithful to the ImageBrief?

Cross-Reference Check, Not Free-Form Review

The approach is to call the LLM again, but this time not to let it improvise freely, but to make it perform a structured judgment against the ImageBrief. The product of this judgment is called PromptInspection:

// Problem solved: Perform structured judgment against ImageBrief, categorize issues + determine severity
export const promptInspectionSchema = z
  .object({
    outcome: z.enum(['block', 'pass', 'revise']),
    issues: z.array(z.object({
      code: z.enum([
        'capability_boundary',    // Requested an unsupported capability (e.g., editing, outpainting)
        'conflict',               // Prompt conflicts with ImageBrief
        'missing_constraint',     // Missed a constraint from ImageBrief
        'missing_subject',        // Missed a subject from ImageBrief
        'unsupported_assumption', // Made an assumption not allowed in ImageBrief
      ]),
      severity: z.enum(['blocking', 'fixable', 'non_blocking']),
    })).max(8),
    revisionInstruction: z.string().max(500).optional(),
  })
  .strict()

Three things are worth clarifying here.

First, the issue classification covers all possible problem dimensions, with no "other." missing_subject and missing_constraint are "omissions," conflict is "contradiction," capability_boundary is "out of bounds," and unsupported_assumption is "hallucination." If the LLM were allowed to classify freely, it would produce unpredictable problem types, and downstream routing could not make deterministic decisions.

Second, the three severity levels directly determine routing. blocking → do not generate; fixable → revise once; non_blocking → do not handle, generate directly. There is no ambiguous space for "possibly severe" or "it depends."

Third, the inspection instruction is "Return no reasoning". The LLM is not allowed to output a chain of thought, only structured judgment. A chain of thought would expose internal execution prompts — users should not see these — and has no value for downstream decisions.

Routing Decision: LLM Judges, Code Decides

How is the inspection result used?

// Problem solved: LLM provides judgment (outcome), code decides routing
export function routeAfterPromptInspection(state: ImageGenerationGraphState) {
  if (state.output?.status === 'failed' || state.output?.status === 'blocked')
    return 'finishBlocked'

  // Only allow one revision if outcome='revise' and no revision has been made yet
  if (state.prompt.inspection?.outcome === 'revise'
      && state.execution.promptRevisionCount === 0)
    return 'revisePrompt'

  // outcome='block' → block; otherwise pass
  return state.prompt.inspection?.outcome === 'block'
    ? 'finishBlocked' : 'finishReady'
}

If the LLM says "revise" but promptRevisionCount is already 1, the code will not give a second chance. If the LLM says "block" but the routing condition is not met, it will not block. The LLM's authority is to "provide judgment," not to "make decisions" — the decision-making power lies in the code.

---

5. Bounded Decisions — Replacing "Try Again" with Hard Counters

The third boundary point: If the prompt is still not good enough after one revision, can it be revised again?

Why a Second Chance is Not Allowed

The more revision opportunities given to the LLM, the more likely it is to get a better prompt. But each revision is an LLM call, consuming tokens and time. More troublesome is the cost of the actual image generation call — if multiple image generations are accidentally triggered during the revision loop, the cost will directly double.

v0.4.12's answer is to seal off this path with three hard-coded limits:

// Problem solved: Hard-coded limits, do not rely on LLM "self-discipline"
export const imageGenerationGraphLimits = {
  maxImageGenerations: 1,      // Call the image generation API at most once (real money)
  maxPlanningModelCalls: 5,    // At most 5 LLM planning calls
  maxPromptRevisions: 1,       // At most one prompt revision
} as const

• maxPromptRevisions = 1: First inspection finds a fixable problem → revise once → second inspection → can only generate or block. There is no "revise again if it's still not good enough after revision."
• maxImageGenerations = 1: Regardless of prompt quality, generate only once. If the result is unsatisfactory, the user needs to re-initiate /image.
• maxPlanningModelCalls = 5: Normal path is 3 calls (brief + draft + inspect), revision path is at most 5 calls (brief + draft + inspect + revise + reinspect). If exceeded, terminate directly without hidden fixes.

Before each node executes, a guard function checks the current count:

export function canRevisePrompt(state: ImageGenerationGraphState) {
  return state.execution.promptRevisionCount
    < imageGenerationGraphLimits.maxPromptRevisions
    && state.output === undefined
}

The LLM has no knowledge of the limits' existence — it is only responsible for outputting structured judgments, and the code decides whether to allow the next step. The three hard limits together boil down to one sentence: Better to generate one fewer time than to generate one more time.

---

6. Security Proxy — Delivering Untrusted External Resources

The fourth boundary point: The image generation API returned a URL, saying the image is here. Can this URL be trusted?

Why the Provider URL Cannot Be Given Directly to the Frontend

The image URL returned by Seedream (the Doubao Seedream text-to-image model, the fixed image model used in this version) is a temporary signed URL pointing to Volcengine's object storage. Giving it directly to the frontend has three problems:

1. The temporary signed URL exposes the Provider's storage domain and signature parameters. Before the signature expires, anyone who gets the URL can access it.
2. Direct frontend access cannot perform ownership verification. If User A guesses User B's runId, they can access User B's image through the URL.
3. It is impossible to perform server-side timeout control, size limits, MIME verification, and redirect interception.

v0.4.12's approach is: The Provider URL is only stored in the server-side database; the frontend only receives a same-origin content path (/api/chat/runs/{runId}/image), which is proxied and read by the server before being returned.

Six-Layer Security Verification Chain

Proxy reading is not a simple fetch + pipe. It is verified layer by layer.

Layer 1: URL Structure Verification

// Problem solved: The URL returned by the Provider is untrustworthy; verify field by field to prevent SSRF
const parsed = new URL(url)
return parsed.protocol === 'https:' &&
  !parsed.username && !parsed.password && !parsed.port && !parsed.hash &&
  isIP(parsed.hostname) === 0 &&
  seedreamImageProviderConfig.resultHosts.includes(parsed.hostname)
  ? url : undefined

Rejects: non-HTTPS, contains username/password, contains port number, contains hash fragment, hostname is an IP address (prevents SSRF internal network probing), hostname is not in the preset allowlist.

Layer 2: Ownership Verification

The server derives ownerSessionHash from the current request's session and compares it with StreamRun.ownerSessionHash in the database. A mismatch directly returns 403.

Layer 3: Status Verification

Must simultaneously satisfy: StreamRun status is completed, ImageGenerationRun providerResultStatus is ready, not expired, not cancelled. Even if the Provider has already generated an image for a cancelled run, the content route permanently denies it.

Layer 4: HTTP Response Verification

When fetching the Provider URL, redirect: 'manual' — rejects redirects. Also checks the Content-Length declaration; if it exceeds 20MB, it is directly rejected.

Layer 5: Streaming Read + Real-time Size Check

// Problem solved: Content-Length header is untrustworthy; must check actual byte count while reading
while (true) {
  const { done, value } = await reader.read()
  if (done) break
  byteLength += value.byteLength
  if (byteLength > maximumImageBytes)  // 20MB
    throw new Error('exceeds allowed size')
  chunks.push(value)
}

Even if the Content-Length declaration is within 20MB, the actual bytes read may exceed it. Check while reading, and terminate immediately if the limit is exceeded. Also set a 15-second timeout — do not wait indefinitely if the Provider storage responds too slowly.

Layer 6: Magic Bytes Verification

// Problem solved: MIME header can be forged; magic bytes are the "ID card" of the file format
function matchesImageMagicBytes(bytes: Uint8Array, mimeType: string): boolean {
  if (mimeType === 'image/jpeg')
    return bytes[0] === 0xFF && bytes[1] === 0xD8 && bytes[2] === 0xFF
  if (mimeType === 'image/png')
    return bytes[0] === 0x89 && bytes[1] === 0x50 && bytes[2] === 0x4E
      && bytes[3] === 0x47 && bytes[4] === 0x0D && bytes[5] === 0x0A
      && bytes[6] === 0x1A && bytes[7] === 0x0A
  // WebP: RIFF....WEBP
  return bytes[0] === 0x52 && bytes[1] === 0x49 && bytes[2] === 0x46
    && bytes[3] === 0x46 && bytes[8] === 0x57 && bytes[9] === 0x45
    && bytes[10] === 0x42 && bytes[11] === 0x50
}

The Content-Type in the HTTP response header can be forged. A response claiming to be image/png could actually contain arbitrary binary data. Magic bytes are the "ID card" of the file format — JPEG must start with FF D8 FF, PNG must start with 89 50 4E 47 0D 0A 1A 0A, WebP must start with RIFF....WEBP. Even if the previous five layers of verification are bypassed, magic bytes can still prevent non-image content from being returned as an image.

What the Frontend Receives

After all six layers of verification pass, the server returns the image bytes to the frontend. The frontend fetches the content path once, generates a Blob, and uses URL.createObjectURL to create a preview address. The same Blob is used for <img> preview and the download button.

Expiration strategy: min(provider expiry, ready + 10min). The expiration time given by the Provider vs. the system's own 10-minute upper limit, whichever is earlier. The frontend clearly prompts in the result area: "Temporary result, please download in time." When the component unmounts, URL.revokeObjectURL releases the Blob.

After refreshing the page, the Blob and object URL no longer exist. v0.4.12 does not promise recovery on refresh — this is intentional, not a missing feature.

---

7. Boundaries and Trade-offs — What Is Not Done Currently and Why

Several things that v0.4.12 explicitly does not do are just as important as what it does.

No Support for Image Editing, Inpainting, Outpainting, or Background Removal

These are not just "text-to-image with an extra parameter." They are each independent capabilities requiring different models or post-processing pipelines. If a user inputs /image remove the background of this image, the system will not silently downgrade to text-to-image — it will return IMAGE_CAPABILITY_UNSUPPORTED and clearly state that only text-to-image is currently supported.

No Support for Generating Multiple Images at Once

Each task generates only one image. Seedream itself supports a group image parameter sequential_image_generation, but this version sets it to disabled. The reason is not a technical limitation, but: under the hard limit of "at most one generation," multiple images would introduce a new decision problem of "which one is the final result," and the cost would be uncontrollable.

---

8. Summary

Walking through an image generation pipeline, the core decisions at the four boundary points can be summarized in a table:

Boundary Point	Core Decision	Consequence of Not Doing This
Structured Extraction	Zod schema + strict + fail-closed	LLM hallucinated fields pass silently; downstream receives untrustworthy data
Quality Inspection	ImageBrief factual anchor + structured inspection judgment	Prompt deviates from user intent; generated result does not match expectations
Bounded Decisions	Hard counters + conditional edge routing	Agent falls into infinite revision loop; cost and latency spiral out of control
Security Proxy	Six-layer verification chain, from URL to magic bytes	Provider URL leaked; SSRF risk; malicious content disguised as an image

The engineering complexity of an AI image generation system is not in "which API to call" — the API formats of Seedream, DALL-E, and Midjourney are largely similar. The real complexity lies in every defense line before and after the API call: how input becomes trustworthy structured data, how prompts are checked, how Agents are constrained, and how external resources are securely delivered.

The boundaries of v0.4.12 are also very clear: single text-to-image, fixed model, no object storage, no HITL, no checkpoint. Each Non-goal leaves a clear extension direction for subsequent versions. If the next version wants to support reference images, multi-image generation, or HITL, the first question to answer is: Which of the existing hard limits and defense lines need to be retained, and which need to be adjusted?

---

Project Link

👉 GitHub: https://github.com/HWYD/ai-mind

👉 Online Experience: https://ai.hwyblog.cloud/instant-mind

If this article or the AI Mind project has been helpful to you, you are also welcome to give the project a Star⭐. This support is very important to me and will give me more motivation to continue organizing the implementation process, design trade-offs, and pitfall reviews of subsequent versions.