Anthropic and OpenAI's Fast LLM Inference Tricks

Anthropic and OpenAI have introduced "fast modes" for their LLMs, employing distinct strategies to achieve faster inference speeds. Anthropic's approach likely involves low-batch-size inference, prioritizing individual user speed at a higher cost. This allows them to serve their actual model (Opus 4.6) without compromising quality. In contrast, OpenAI's fast mode utilizes GPT-5.3-Codex-Spark, a less capable model, and is backed by Cerebras chips, enabling significantly higher speeds. The tradeoff is a reduction in model fidelity. The choice between these approaches depends on the specific application and the relative importance of speed versus accuracy. Anthropic's method is analogous to a bus that leaves immediately when a passenger boards, while OpenAI's is like using a faster, but less intelligent, robot brain. The AI labs are not fully transparent about the details of their fast modes, but the author's analysis provides a plausible explanation based on available information and industry knowledge.

Transparency Disclosure: This analysis was prepared by an AI language model, Gemini 2.5 Flash, based on information from the provided article. Human oversight ensures adherence to journalistic standards and legal compliance, including EU AI Act Art. 50.