← Back to the summary

How catbuddy Survives Three LLM APIs, Rate Limits, and Truncated Output Without Crashing

This is the 8th article in the "Agent Full-Stack Development in Practice" series. The entire series uses catbuddy (a local-first AI programming assistant, approximately 36,000 lines of TypeScript) as a case study, deconstructing harness design from shallow to deep.

In the previous articles, we finished explaining the "kernel" of the harness—the heart, hands and feet, eyes, and memory. This article starts discussing resilience: your Agent runs in the real world, facing three APIs with different formats, networks that rate-limit and jitter, and models that return empty strings or get truncated. This article explains how catbuddy survives in this chaos—one interface handling the differences of three providers, automatic failover when the primary model goes down, and self-healing from transient errors in a single call. After reading, you can explain to your colleagues "how many layers a production-grade Agent's fault tolerance has and what each layer manages."

0. An Afternoon That Made Me Crash from Switching Models

The first time I connected DeepSeek to catbuddy, I was full of confidence—just calling one more API, right?

It exploded as soon as I ran it. Claude's system prompt is a top-level parameter, while DeepSeek, following OpenAI, treats system as one message in the message array; Claude's tool result role is user, DeepSeek's is tool; Claude's streaming response uses SSE events, DeepSeek ends with data: [DONE]... Every time I connected a new provider, I had to add another if (provider === 'xxx') in AgentRunner, making the code look more and more like spaghetti.

Worse was yet to come. Even after all three were connected, the real world would still stab you in various ways: DeepSeek rate-limiting at midnight (429), Anthropic occasionally returning 529 overloaded, the model sometimes giving me an empty string, writing a long file only to be cut off halfway by max_tokens... If any one of these wasn't caught, a 40-turn session would "pop" and disappear entirely, leaving the user with just an An error occurred message.

This article is the answer I redesigned after that afternoon. It's divided into three sections, from outside to inside:

Adaptation Layer—a single LLMProvider interface that welds the differences of the three APIs shut inside subclasses, so AgentRunner has no idea who's underneath;
Failover—when the primary model goes down, automatically tries the fallback model chain, and AgentRunner remains completely oblivious;
Three-Layer Retry—transient errors in a single call (timeout / empty reply / truncation), each layer swallows what it can handle and passes up what it can't.

To sum it up in one sentence: First, ensure one provider's failure doesn't affect others; then, let others take over; finally, let each call withstand minor turbulence on its own.

1. Adaptation Layer: One Interface, Welding Shut Three Providers' Differences

1.1 Core Contract: AgentRunner Only Recognizes One Interface

In the entire harness, there is only one place that actually calls the LLM, which is provider.chatStreamWithRetry(). When AgentRunner calls it, it has no idea whether it's asking Claude or DeepSeek, nor whether a failover has already occurred.

The core of this abstraction is the LLMProvider abstract base class (providers/base-provider.ts)—it defines the unified interface and collects all logic shared by all Providers in the base class (retry, role alternation enforcement, error classification); the two subclasses each implement the actual chatStream():

// providers/base-provider.ts
export abstract class LLMProvider {
  abstract readonly name: string

  abstract chat(opts: ChatStreamOpts): Promise<LLMResponse>       // Non-streaming
  abstract chatStream(opts: ChatStreamOpts): Promise<LLMResponse> // Streaming · Subclass must implement

  // ⭐ Streaming call with retry —— Template method, written in base class, subclasses don't need to override
  async chatStreamWithRetry(opts: ChatStreamWithRetryOpts): Promise<LLMResponse> { /* ... */ }

  protected isTransientError(response: LLMResponse): boolean { /* ... */ } // Error classification
  protected enforceRoleAlternation(messages: LLMMessage[]): LLMMessage[] { /* ... */ } // Role alternation
}

This is the textbook Strategy Pattern—Provider is the replaceable "strategy," AgentRunner is the "context." In other words: the base class manages "things that are the same for all providers" (how to retry, what counts as a transient error, the tail can't be an assistant), and the subclasses only manage "the dirty work unique to this provider" (message format, how to assemble streaming). The whole picture looks like this:

Note the bottom right corner: the "Compat" in the name OpenAICompatProvider is not filler—any service following the OpenAI chat/completions protocol goes through it: DeepSeek, Ollama, vLLM, OpenRouter, Tongyi, Moonshot... one class covers half the AI ecosystem. The routing logic in the factory function (providers/factory.ts) is therefore simple to the point of being unreasonable:

// providers/factory.ts —— Just look at apiBase to know which path to take
function makeProvider({ model, apiKey, apiBase }): LLMProvider {
  if (apiBase?.includes('anthropic')) {
    return new AnthropicProvider({ apiKey, apiBase, defaultModel: model })
  }
  return new OpenAICompatProvider({ apiKey, apiBase, defaultModel: model }) // Everything else goes through the compatibility layer
}

This isn't laziness; it's an acknowledgment of the reality that "OpenAI-compatible API has become a de facto standard."

1.2 Where the Three Providers Actually Differ: Three Hard Wounds in Message Format

The heaviest dirty work the subclasses do is message format conversion. The upper layer always uses the unified LLMMessage (where tool results are honestly role: 'tool'), and converting them into the format recognized by each API is the adapter's job. The three most likely differences to cause crashes:

Difference Point	Anthropic	OpenAI Compatible (incl. DeepSeek)
system prompt	Top-level parameter `system: "..."`	A message `{ role: 'system' }` in the message array
Tool result role	`role: 'user'` + `tool_result` block	`role: 'tool'` + `tool_call_id`
Tool calls in streaming	SDK event callbacks, assembled and handed to you directly	Parameters are split into multiple chunks, need to be assembled by index yourself

For the first and second points, the Anthropic adapter performs extraction and wrapping when building the request: system is filtered out from the messages and stuffed into the top-level parameter, and messages with role: 'tool' are rewritten into a tool_result content block inside a role: 'user' message (the Claude API simply doesn't recognize role: 'tool'). All of this is encapsulated inside the adapter; AgentRunner knows nothing.

The third point is the most insidious—OpenAI's streaming will split the parameters of a tool call into several chunks, each chunk carrying only a small snippet of the JSON string. You can't just JSON.parse a chunk upon receiving it; you have to accumulate them by index before parsing:

// providers/openai-compat.ts —— Aggregate streaming tool_calls by index
const toolCallMap = new Map<number, { id: string; name: string; args: string }>()
for await (const chunk of stream) {
  for (const tc of chunk.choices?.[0]?.delta?.tool_calls ?? []) {
    if (!toolCallMap.has(tc.index)) toolCallMap.set(tc.index, { id: '', name: '', args: '' })
    const entry = toolCallMap.get(tc.index)!
    if (tc.id) entry.id = tc.id
    if (tc.function?.name) entry.name = tc.function.name
    if (tc.function?.arguments) entry.args += tc.function.arguments  // ⭐ Streaming concatenation
  }
}
// Only after the loop ends do we safeParseJSON(entry.args) —— Accumulate fully then parse

By the way, a little trick for "capability detection": although the format is compatible, the features may not be. DeepSeek doesn't support image input. catbuddy detects this using apiBase.includes('deepseek.com') and automatically downgrades images to a text placeholder ([N attached image(s) not sent: this provider only accepts text...]), avoiding a baffling 400 error. The cost of one class covering half the ecosystem is having to leave a few switches for these "dialects."

1.3 A Hidden Pitfall That Took Me an Afternoon to Debug: DeepSeek's reasoning_content Must Be Carried Back and Forth

This is the highlight of this section, and also where I was truly stuck for hours that day.

Reasoning models like DeepSeek R1 will spit out an extra reasoning_content in the streaming response—its "thinking process." This is not a standard OpenAI field; it's a private addition DeepSeek stuffed into the OpenAI format. Capturing it is simple, just like capturing the main text content, by handling one more delta:

// providers/openai-compat.ts · chatStream()
const reasoningDelta = (delta as { reasoning_content?: string }).reasoning_content
if (reasoningDelta) {
  reasoningContent += reasoningDelta
  await opts.onThinkingDelta?.(reasoningDelta)   // Streams all the way to the collapsible 'reasoning bubble'
}
// ...finally returned along with LLMResponse
return { content, toolCalls, /* ... */ reasoningContent: reasoningContent || undefined }

Up to this point, everything was smooth. The pitfall lies in multi-turn tool calls.

Think about this scenario: the model first "thinks" for a while, then decides to call a tool (tool_use). We execute the tool, feed the result back, and let it continue thinking about the next step. The problem is: during the second call, what happens if that previous round's reasoning_content is not included?

It will talk nonsense. In multi-turn tool calls, DeepSeek relies on its own reasoning chain from the previous round to maintain coherent thinking. If you erase its "scratch paper" from the previous round, it can't pick up the thread—at best, it repeats work; at worst, it gives conclusions that contradict the context above. My symptom that day was: single-turn Q&A was fine, but once continuous tool calls were involved, the model's answers were completely incoherent.

The solution is to make reasoning_content travel back and forth with the conversation, in two phases:

SAVE Phase (Persistence): The reasoningContent returned by the model in this round must be saved by AgentRunner into this assistant message (along with tool_use, it falls into the history and also onto the disk).
BUILD Phase (Retrieve and Refill): When building the request for the next round, the adapter re-attaches the stored reasoning_content back onto this assistant message before sending it out.

The code on the BUILD side is hidden in toOpenAIMessages, just a few lines, but missing it causes bugs:

// providers/openai-compat.ts · toOpenAIMessages()
if (m.role === 'assistant') {
  const hasTools = (m.toolCalls?.length ?? 0) > 0
  const reasoning = m.reasoningContent ?? ''
  if (hasTools || reasoning) {
    row.reasoning_content = reasoning   // ⭐ Bring the previous round's reasoning chain back as is
  }
}

Drawn as a diagram, this "round trip" is clear at a glance:

Just remember one sentence: For models like DeepSeek, the reasoning chain is part of the conversation state; lose it, and it talks nonsense. This is also the most hidden cost of "one interface handling three providers"—you think it's just format conversion, but actually, some "dialects" require you to manage their state too.

2. Failover: Primary Model Down, Auto-Switch, AgentRunner Unaware

The adaptation layer solved "which provider to call and how." The next question: What if the entire primary model goes down? DeepSeek rate-limiting at midnight, Anthropic 529 overloaded—you can't just leave the user waiting.

catbuddy's answer is FallbackProvider (providers/fallback.ts). The user configures a "primary model" + a fallbackModels list, and the factory function strings them into a chain. The essence here is: FallbackProvider itself is also an LLMProvider—it inherits the same base class and exposes the same interface externally. This is the Decorator Pattern: the outer layer wraps the logic of "try primary first, then fallback if it fails," but what AgentRunner receives is still the familiar LLMProvider; it has no idea whether what it holds is a single model or a failover chain.

2.1 How the Factory Strings the Chain Together

// providers/factory.ts
export function createProvider(config): LLMProvider {
  const primary = buildProvider(config, defaults.model, defaults.provider)

  const fallbackModels = config.agents.defaults.fallbackModels ?? []
  if (fallbackModels.length === 0) return primary   // No fallback configured, return bare primary directly

  const fallbacks: LLMProvider[] = []
  for (const fb of fallbackModels) {
    try {
      fallbacks.push(buildProvider(config, /* resolve from preset name or inline config */))
    } catch (err) {
      console.warn(`[factory] Fallback initialization failed, skipping: ${err.message}`)  // Swallow, don't block others
    }
  }
  if (fallbacks.length === 0) return primary
  return new FallbackProvider({ primary, fallbacks })  // ⭐ Decorate one layer, externally still LLMProvider
}

fallbackModels supports two formats: a string (referencing a preset name in modelPresets, like "fast"), or an inline object ({ model: 'gpt-4o', provider: 'openai' }). If a fallback fails to initialize (e.g., no API Key configured), the catch directly swallows and skips it, without affecting others—use as many as you can.

2.2 FallbackProvider: Try in Order, Always Give a Meaningful Fallback

// providers/fallback.ts · _tryWithFallback()
const providers = [this.primary, ...this.fallbacks]
for (let i = 0; i < providers.length; i++) {
  const provider = providers[i]
  try {
    const response = await provider[method]({ ...opts, model: opts.model ?? provider.defaultModel })
    if (response.finishReason !== 'error') {
      if (i > 0) console.log(`[fallback] Switched to fallback #${i}: ${provider.name}`)
      return response                      // ✅ This one succeeded, return immediately
    }
    continue                               // ❌ This one errored → try the next one directly
  } catch (err) {
    console.warn(`[fallback] ${provider.name} threw exception: ${err.message}`)
    continue
  }
}
// All providers failed
return { content: 'All providers failed. Please check your API keys and network connection.',
         finishReason: 'error', /* ... */ }

The entire chain drawn out is a "step-by-step degradation" waterfall:

There are two design judgments here I particularly want to emphasize:

One: Don't get hung up on "transient vs. permanent," just try if you can. You might think you should only switch to the next provider on "transient errors" (rate limiting, 5xx). But catbuddy's strategy is: even if this provider reports a seemingly "permanent" error like quota exhaustion, still try the next one. The reason is practical—isTransientError judgments are inherently heuristic and unreliable; moreover, your budget/quota situation on different providers might be completely different. The cost of trying one more provider is far lower than missing a provider that could have worked.

Two: FallbackProvider itself overrides chatStreamWithRetry to avoid retry explosion. Note: the chain switching uses each provider's chatStream, not each provider's chatStreamWithRetry. Why? Otherwise, it becomes an exponential explosion of "primary model retries 3 times × fallback retries 3 times = 9 calls." Failover manages "switching people," single-machine retry manages "trying the same person again"—these two things must be separated, not multiplied.

In the worst-case scenario, that sentence All providers failed. Please check your API keys and network connection.—it's not a casually written fallback. It clearly states three things in one sentence: what happened (call failed), possible reasons why (Key or network), and what you can do (go check). Compared to a completely blank white screen, this message at least gives the user a next step. This is the iron rule running through the entire resilience design: Never crash in front of the user.

2.3 Hot Swap: Behind the `/model` Command, How the Provider Switches Without Restarting

Failover is "passive" model switching. There's also "active" switching—the user types /model gpt-4o in the chat box.

(The three-layer routing of the /model command itself was covered in Article 03; here we only look at the hot swap effect it triggers.)

The core requirement for hot swapping is demanding: The Agent cannot restart, cannot interrupt the current conversation, but the next round must use the new model. Moreover, changing one model requires synchronously notifying a chain of subsystems—AgentRunner, memory merger Consolidator, background learning Dream, sub-agents—they are all using the same set of Providers.

Here's a key optimization: model_presets.ts calculates a Signature for each configuration—essentially a hash of "model name + key configuration" (in implementation, it's a tuple like ['model_preset', presetName, JSON.stringify(presetConfig)]). When switching, first compare Signatures:

// agent/loop.ts · _applyProviderSnapshot()
const sig = JSON.stringify(snapshot.signature)
if (sig === prevSig) return   // ⭐ Configuration unchanged, do nothing, absolutely don't new another Provider

// Really changed → then broadcast to all subsystems
this.provider = snapshot.provider
this.runner.setProvider(snapshot.provider)               // Agent executor
this.consolidator.setProvider(snapshot.provider, /*...*/) // Memory merge
this.dream?.setProvider(snapshot.provider, /*...*/)       // Background learning
this.subagents?.setProvider(snapshot.provider, /*...*/)   // Sub-agents

If the Signature matches, it returns directly, avoiding the waste of "re-initializing the Provider even though the configuration hasn't changed" (rebuilding a Provider requires reconnecting the SDK and re-detecting capabilities, which isn't cheap). The entire switch is fully synchronous, zero restart, zero interruption—the currently running turn finishes using the old Provider, and the next turn automatically uses the new one. AgentRunner.setProvider(newProvider), one line, done.

3. Three-Layer Retry: How Transient Errors in a Single Call Self-Heal

Failover solves the problem of "the entire model is unavailable." But more frequent are minor issues in a single call: a network hiccup, the model returning an empty string, output being cut off halfway by truncation. These don't require switching models; self-healing on the spot is enough.

catbuddy lays down three lines of defense here, with a key design principle: Each layer only swallows errors it can handle; what it can't handle is passed up as is.

Note the "positions" of these two layers: ① Provider retry is at the bottom layer, close to the API; ③ AgentRunner self-recovery is at the outermost layer, because only after running a complete call and getting the full response do you know if the model "returned an empty string" or was "truncated." In between, there's actually the Fallback chain from Section 2 ("Can't answer and retries are ineffective → switch model"). The three are nested layer by layer, from inside to outside: single-machine retry → switch model → content-level remediation.

3.1 First Layer: Provider Retry—Only Retry "What Should Be Retried," Exponential Backoff

The bottom layer is chatStreamWithRetry (written in the base class, shared by all Providers). It does two things: exponential backoff + only retry transient errors.

// providers/base-provider.ts
async chatStreamWithRetry(opts): Promise<LLMResponse> {
  const maxAttempts = opts.retryMode === 'persistent' ? Infinity : 3
  const baseDelays = [1, 2, 4]  // 1s → 2s → 4s

  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    try {
      const response = await this.chatStream({ /* ... */ })
      if (response.finishReason !== 'error') return response   // Success
      if (!this.isTransientError(response)) return response     // Permanent error, retrying is useless, return directly

      const delay = response.retryAfter ?? baseDelays[Math.min(attempt, 2)]  // 429 might come with retryAfter
      await sleep(delay * 1000)
    } catch (err) {
      if (err.name === 'AbortError') throw err   // ⭐ User /stop, penetrate immediately, never wait for backoff
      await sleep(baseDelays[Math.min(attempt, 2)] * 1000)
    }
  }
  return { content: 'Error: max retries exceeded', finishReason: 'error', /* ... */ }
}

Three key points:

① "Should it retry?" is decided by isTransientError , and it's restrained:

protected isTransientError(response: LLMResponse): boolean {
  if (response.errorShouldRetry !== undefined) return response.errorShouldRetry
  if (response.errorStatusCode === 429) return true                  // Rate limit → wait and it'll pass
  if (response.errorStatusCode && response.errorStatusCode >= 500) return true  // Server fault → trying another instance might work
  if (response.errorKind === 'timeout' || response.errorKind === 'connection') return true // Network jitter → reconnect
  return false  // 401 Wrong Key, 400 Wrong Format → retrying 100 times is useless, don't waste tokens
}

429 (Rate Limit) gets special treatment: besides being judged as transient and retryable, it will also prioritize reading retryAfter from the response—the server explicitly says "come back in X seconds," so wait X seconds instead of stupidly waiting a fixed 1/2/4. For 4xx (except 429) client errors, never retry—if your Key is wrong or the format is wrong, retrying more times yields the same result and might push you toward stricter rate limits.

② AbortError penetrates directly, never retried. The user clicked /stop, AbortController triggered. If you're still stuck in backoff waiting 4 seconds at this point, the user will think "I pressed /stop three seconds ago and it's still spitting out text." AbortError is not an "error"; it's "the user wants to stop"—highest priority, immediately penetrates all retry layers.

③ Two retry modes: standard (default, max 3 times) for normal user conversations—if three times still don't work, there's probably a real problem, and waiting more is meaningless; persistent (no upper limit) for background tasks, like Dream refining memories in the background or Consolidator compressing history—no one is waiting, so trying a few more times doesn't matter. But regardless of the mode, AbortError always penetrates.

3.2 Second Layer: Model "Responded, but Responded with Nothing"—Empty Reply Retry

The Provider layer manages "can the model respond." But sometimes the model responds, yet gives you an empty string—no text, no tool_calls, nothing at all. Usually, it "got stuck," or the previous tool result confused it.

This kind of thing can't be detected by the Provider layer (from its perspective, this call "succeeded"), so AgentRunner has to catch it in the loop:

// agent/runner.ts
if (!finalContent?.trim() && emptyRetries < MAX_EMPTY_RETRIES) {
  emptyRetries++
  messages.push({
    role: 'user',
    content: 'Please provide your response to the user based on the conversation above.',
  })
  continue   // Push a guiding message, go another round
}

No error is thrown; instead, a "Please provide your response based on the conversation above" is inserted, letting it try again. Up to 2 times.

3.3 Third Layer: Output Cut Off Midway by `max_tokens`—Continuation Recovery

The last type: the model is writing a long file, hits the max_tokens limit halfway, finishReason becomes max_tokens, and the sentence is cut off mid-speech.

AgentRunner detects this signal and sends a "continue writing" prompt to let it pick up:

// agent/runner.ts
if (response.finishReason === 'max_tokens' && lengthRecoveries < MAX_LENGTH_RECOVERIES) {
  lengthRecoveries++
  messages.push({
    role: 'user',
    content: 'Output limit reached. Continue exactly where you left off — no recap, no apology.',
  })
  continue   // The continuation content will be appended after the previous output
}

Up to 3 times—if it hasn't finished writing after more than 3 times, the model is likely stuck in an infinite repetition loop, and continuing is a waste.

The wording of this guiding message was polished repeatedly: no recap, no apology. Without this addition, the model's typical reaction is: "Okay, let me continue. First, let me review what I did before..."—and then half the token budget is spent on recapping, with hardly any real work done. One precise prompt saves half the tokens. This is the same as the empty reply's "Please provide your response based on the conversation above"—both are using prompts to pull the model back from a wrong path.

By the way: when tool execution throws an exception, AgentRunner also doesn't crash; instead, it wraps the exception as Error: ${err.message} and feeds it back to the model as a tool result—letting the model see the error itself and change parameters or tools on its own. The code only faithfully reports; the decision-making power is left to the LLM. This line belongs to the tool system, covered in detail in Article 04, so I won't expand on it here.

4. Connecting the Three Segments: How a Message Is Caught Layer by Layer

Putting the three segments back together, a call that narrowly avoids disaster looks like this (most intuitive from the logs):

[run] Calling provider.chatStreamWithRetry()
[run] [fallback] primary (deepseek) reported 429 rate limit ...        ← Failover layer catches it
[run] [fallback] Switched to fallback #1: anthropic/claude-sonnet  ← Switched to fallback, success
[respond] [DONE] turn completed {totalMs: 14000}            ← User completely unaware throughout

AgentRunner only called chatStreamWithRetry once from start to finish. As for whether there were retries in between, or a model switch, or a continuation—it has no idea, nor does it need to. This is the fulfillment of the main thread from Article 01 in the "resilience" layer: each organ only recognizes the interface, not the implementation; changes are locked inside a single organ. Add a new provider, AgentRunner doesn't change a single line; tweak the retry strategy, code outside the Provider is completely unaware.

What did this article cover?

Adaptation Layer: The LLMProvider abstract base class collects common logic like "retry, role alternation, error classification" in the base class; the two subclasses AnthropicProvider / OpenAICompatProvider each implement chatStream() to digest the differences of the three providers (system as parameter vs. message, tool result role, streaming tool_call assembly). AgentRunner only calls chatStreamWithRetry(), never knowing who's underneath. The most hidden pitfall: DeepSeek's reasoning_content must be SAVEd and then refilled during BUILD in the next round, otherwise multi-turn tool calls will talk nonsense.
Failover: FallbackProvider uses the Decorator Pattern to string the primary model + fallbackModels into a chain, externally still the same interface; "try if you can" doesn't get hung up on error types, and if all fail, gives a meaningful fallback message. /model hot swapping relies on Signature comparison to avoid redundant initialization; the current turn finishes, the next turn automatically switches, zero restart.
Three-Layer Retry: ① Provider layer exponential backoff (1s/2s/4s, max 3 times, special handling for 429, AbortError penetration); ② Empty reply injects guiding message (max 2 times); ③ max_tokens truncation injects "continue, no recap" continuation (max 3 times). Each layer only swallows what it can handle, passing up what it can't—the core iron rule is "never crash in front of the user."

Next Article Preview: These three layers of retry are hardcoded in the loop. But what if I want to add monitoring before and after tool execution, connect Langfuse tracing, or insert some project-specific logic—I can't go modify AgentRunner's source code every time, right? The next article will explain how catbuddy uses a Hook system to open holes at key nodes in the loop, allowing you to plug in extensions without touching a single line of core code.