Google, Red Hat, and IBM Ship llm-d 1.0 Open-Source AI Serving Stack

On October 20 at the PyTorch Conference, Google, Red Hat, and IBM shipped llm-d 1.0, a free, vendor-neutral CNCF Sandbox serving stack that provides the KV cache system, prefill/decode disaggregation, and load balancing that DeepSeek withheld from its open-source production engine. Founded in 2025 by Red Hat, IBM, Google Cloud, CoreWeave, and NVIDIA with contributions from more than 500 developers across over 25 organizations, the release orchestrates above model servers like vLLM and SGLang to provide multi-tenant flow control, zero-downtime upgrades, and session-aware routing across different GPU vendors. According to vendor-reported figures without independent verification, IBM Research used open frontier models on llm-d at an 11.5x lower inference cost than an all-proprietary setup with a roughly 98% prefix-cache hit rate and a blended cost of about $0.11 per million tokens, while a separate test on an approximately 753B-parameter GLM-5.2 model reached over 6.6M output tokens per minute for up to 3,000 concurrent coding agents at 5x to 10x lower cost per token than equivalent commercial APIs. Additional vendor-reported production metrics include Google Cloud doubling Gemini prefix cache hit rates from 35% to 70% while cutting time-to-first-token by over 35% for coding agents, AWS seeing up to 70% more tokens per second on SageMaker HyperPod, Tesla and Red Hat reporting 3x output throughput on Llama 3.1 70B, and Snap reaching 75% to 80% prefix cache hit rates. "The bottleneck shifts from generating answers to managing the economics, operational reliability, and extreme infrastructure demands of complex agentic workloads." — Brian Stevens, Red Hat