跪拜 Guibai
← All articles
AI Programming

How catbuddy Survives Three LLM APIs, Rate Limits, and Truncated Output Without Crashing

By 浪遏 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

Most agent demos paper over the reality that LLM APIs fail in incompatible, provider-specific ways. catbuddy's design shows how to absorb those differences behind one interface, chain fallback models without the runner knowing, and recover from empty or truncated responses with targeted prompt injections—patterns that turn a brittle prototype into something that can run unattended.

Summary

A single abstract LLMProvider base class isolates every API quirk—system prompts as top-level parameters versus messages, tool-result role mismatches, and streaming tool-call chunk assembly—behind two concrete adapters. An OpenAICompatProvider handles any OpenAI-compatible service, while an AnthropicProvider translates Claude's dialect. The trickiest detail is DeepSeek's reasoning_content: the reasoning chain must be saved from one turn and replayed on the next, or multi-turn tool calls derail into nonsense.

Above the adapters, a FallbackProvider wraps a primary model and a list of fallbacks into a single decorator that tries each in sequence. It does not distinguish transient from permanent errors before switching; the cost of skipping a usable provider outweighs the cost of an extra attempt. A hot-swap mechanism triggered by the /model command compares configuration signatures to avoid reinitializing providers when nothing changed, letting the next turn pick up the new model without restarting the agent.

Inside a single call, three retry layers catch progressively higher-level failures: exponential-backoff retries at the provider level for 429s and 5xx errors, empty-response injection that nudges the model to try again, and max_tokens truncation recovery that tells the model to continue exactly where it left off without recapping. Every layer swallows only the errors it can fix and passes the rest upward, so the user never sees a blank crash screen.

Takeaways
— An abstract LLMProvider base class holds shared retry, error-classification, and role-alternation logic; subclasses implement only provider-specific chatStream formatting.
— OpenAICompatProvider covers DeepSeek, Ollama, vLLM, OpenRouter, and any service that speaks the OpenAI chat/completions protocol.
— DeepSeek's reasoning_content must be persisted on the assistant message and re-sent on the next request, or multi-turn tool-calling coherence breaks.
— FallbackProvider is itself an LLMProvider, so AgentRunner never knows whether it is talking to a single model or a failover chain.
— Fallback tries every provider in sequence regardless of whether the error looks transient or permanent; a heuristic is less reliable than simply attempting the next available model.
— Hot-swapping models via /model compares a configuration signature to skip reinitialization when nothing changed, then swaps the provider for the next turn without restarting.
— Provider-level retry uses exponential backoff (1s, 2s, 4s) and respects server-supplied retryAfter headers on 429 responses; AbortError from a user /stop command penetrates immediately.
— Empty responses trigger an injected user message asking the model to reply based on the conversation; max_tokens truncation triggers a continuation prompt that explicitly forbids recapping.
— Every resilience layer only handles the errors it can fix and passes the rest upward, ensuring the user never sees a raw crash.
Conclusions

Treating the OpenAI chat/completions format as a de-facto standard and routing everything except Anthropic through a single compat provider is a pragmatic bet that pays off in reduced maintenance surface.

The reasoning_content round-trip requirement for DeepSeek reveals that API compatibility is not just about wire format—some models leak state into the conversation that the adapter must manage.

FallbackProvider's refusal to classify errors before switching is a useful counterpoint to the common instinct to gate failover on transient-error detection; the heuristic is often wrong, and the cost of an extra call is low.

Hardcoding 'no recap, no apology' into the truncation-recovery prompt is a small but high-leverage optimization: without it, models burn tokens summarizing what they already wrote instead of continuing.

AbortError penetration through all retry layers is a UX requirement, not a technical one—users who hit stop expect the agent to stop immediately, not after a backoff timer expires.

Concepts & terms
LLMProvider abstract base class
A shared interface in catbuddy that defines chat and chatStream methods, plus template methods for retry logic, error classification, and role-alternation enforcement. Concrete providers like AnthropicProvider and OpenAICompatProvider only implement the actual API call formatting.
OpenAICompatProvider
A single adapter that handles any LLM service conforming to the OpenAI chat/completions protocol—DeepSeek, Ollama, vLLM, OpenRouter, and others—by translating catbuddy's internal message format into OpenAI-compatible requests.
reasoning_content round-trip
DeepSeek R1 emits a reasoning_content field during streaming that must be saved onto the assistant message and re-sent on the next request; without it, the model loses its chain of thought across multi-turn tool calls and produces incoherent output.
FallbackProvider (Decorator Pattern)
A provider that wraps a primary LLMProvider and a list of fallback providers, exposing the same LLMProvider interface. It tries each in sequence on failure, so the caller never knows a failover occurred.
Signature-based hot swap
When the /model command changes the active model, catbuddy computes a hash of the model name and key configuration. If the signature matches the current one, it skips reinitialization; otherwise it swaps the provider for the next turn without restarting the agent.
Three-layer retry
Provider-level exponential-backoff retry for transient HTTP errors, empty-response injection that nudges the model to reply, and max_tokens truncation recovery that prompts the model to continue without recapping. Each layer only handles the class of error it can fix.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗