tokenspeed

Why we included this project

TokenSpeed is an LLM inference engine built for agentic workloads, where many small sequential requests and long context put real pressure on a serving stack. It claims TensorRT-LLM-level performance with a vLLM-like interface, so teams used to OpenAI-compatible serving can adopt it without rewriting their clients. The engineering stands out: a static compiler derives collective communication from module annotations, removing the need to hand-write parallelism, and a C++ control plane encodes request lifecycle and KV cache ownership as a typed state machine, which makes safe memory reuse under load more predictable. It also includes a fast MLA kernel tuned for Blackwell, which matters for anyone running DeepSeek, Kimi, Qwen, or similar latent-attention models on recent NVIDIA GPUs. For latency-sensitive production, this is worth benchmarking against your own traffic.