TokenSpeed
LLM serving engine tuned for agentic workloads on datacenter GPUs
What it is
An inference engine for serving large language models, built around a C++ scheduler that models each request and its KV cache usage as a finite-state machine, plus a Python execution layer that runs the model code. Kernels sit in a separate pluggable registry, including a fast Multi-head Latent Attention implementation for NVIDIA Blackwell, with support for AMD Instinct GPUs as well. Teams running models like DeepSeek, Kimi, Qwen, and GLM in production use it to get high per-user token throughput for agent-style traffic.