SageAttention

Why we included this project

Inference on modern transformers often comes down to how fast the attention kernel runs, and SageAttention targets exactly that piece. It replaces the standard attention computation with quantized CUDA and Triton kernels tuned for Ampere, Ada, Hopper, and Blackwell GPUs, giving roughly 2 to 5 times the throughput of FlashAttention while end-to-end quality stays intact for language, image, and video models. You can swap in a drop-in function that applies kernels tuned for your GPU generation, so integrating it into an existing pipeline is fairly low-effort. The maintainers keep several versions available, which is useful: SageAttention2 is the precision-first option, while the newer variants push for more speed. Combined with support for torch.compile and distributed inference.