Cache-Aware Prefill-Decode Disaggregation Boosts LLM Serving Speed by 40%

The article discusses cache-aware prefill-decode disaggregation (CPD), a serving architecture developed by Together AI to improve the performance of long-context LLM inference. The key idea behind CPD is to separate cold and warm requests based on their cache hit rate. Cold requests, which contain mostly new context, require full computation, while warm requests, which contain large portions of context that have been seen before, can leverage cached information. By handling these two types of requests with separate compute resources, CPD optimizes resource allocation and reduces latency. The architecture leverages distributed KV cache for fast context reuse, enabling faster time-to-first-token (TTFT) and higher sustainable throughput. The article highlights that CPD improves sustainable QPS by up to 35-40% over existing disaggregated designs. This improvement is particularly significant under mixed, real-world traffic conditions, where both cold and warm requests are present. CPD addresses the challenges posed by long-context inference, where TTFT can increase and become more variable. By optimizing resource allocation and reducing latency, CPD enables more responsive and scalable LLM deployments. However, implementing CPD requires significant engineering effort and infrastructure investment. The complexity of the architecture may also introduce new challenges in terms of monitoring and maintenance. Further research and development are needed to explore the full potential of CPD and to address its limitations.