native-sparse-attention-pytorch

Why we included this project

Transformer training costs grow quadratically with context length, and this library is a direct, pip-installable answer: it reproduces the sparse attention scheme DeepSeek laid out in their Native Sparse Attention paper. You instantiate a SparseAttention module, set a few hyperparameters for the sliding window and the compression and selection blocks, and swap it in where you would normally use full attention. Because the kernels are built on Triton and PyTorch's Flex Attention rather than hand-written CUDA, the code doubles as a readable reference for prototyping hardware-aligned sparse attention. The bundled Enwik8 language-modeling example trains the module end to end, so teams testing long-context efficiency can see it work before committing to a production kernel.