sglang

SGLang serves large language models in production with the kind of throughput numbers that make vLLM look conservative. The project reports up to 5x faster inference on general models and 7x on DeepSeek's MLA architecture. It powers an unusual cross-section of the industry: xAI, AMD, NVIDIA, LinkedIn, and the major cloud providers all run it, reportedly across more than 400,000 GPUs.

Setup is the standard NVIDIA inference stack: CUDA, drivers, docs.sglang.io, and a chassis full of GPUs. Supported hardware spans NVIDIA GB200/H100/A100, AMD MI355/MI300, Intel Xeon, Google TPUs, and Ascend NPUs. Models include Llama, Qwen, DeepSeek, GLM, Gemini, Mistral, and most Hugging Face models. The framework is OpenAI-API compatible so existing clients drop in.

Solo and small teams running open-weight models: this is one of the strongest options on the shelf, especially if you're on DeepSeek or running heavy agentic workloads. Large teams running production inference at scale: you're probably already evaluating it. The 400K-GPU adoption number is not marketing; xAI and LinkedIn deployments are real.

The catch: serious production-grade inference is still serious work. Cold starts, KV cache tuning, and multi-node setups need real engineering. SGLang gives you a faster engine; it doesn't remove the operational burden of running an inference platform.