TurboTransformers

Why we included this project

TurboTransformers comes from Tencent's WeChat AI team and has already run real online BERT services inside the company, so the speedups it claims come from actual production workloads rather than benchmarks alone. It covers BERT, ALBERT, RoBERTa, and GPT-2 style encoders and decoders on CPU and GPU, and it slots into existing PyTorch code as a small plugin instead of forcing a rewrite of the serving path. Variable-length inputs need no offline graph preprocessing, so you can change batch size and sequence length at runtime, and smart batching keeps zero-padding waste low when requests come in different lengths. In production Tencent reports 1.88x faster WeChat FAQ matching, 2.11x faster sentiment analysis on its public cloud, and 13.6x faster QQ recommendation. Both Python and C++ APIs are provided, which matters if you want to embed inference in a non-Python stack.