Liger-Kernel

Why we included this project

Teams fine-tuning or pretraining large language models on multiple GPUs will find this library useful because it swaps the heaviest Transformer layers for tightly fused Triton kernels. Swapping in its implementations of RMSNorm, RoPE, SwiGLU, cross-entropy, and fused linear-cross-entropy typically lifts throughput on the same Hugging Face training run by roughly 20 percent while cutting GPU memory use noticeably. That freed memory often goes toward longer context windows, bigger batch sizes, or fitting on smaller hardware instead of renting more nodes. The project also ships optimized kernels for preference-alignment losses like DPO, ORPO, and SimPO, which tend to hit memory bottlenecks. Integration is straightforward: a one-line patch covers a supported model, or you can compose your own using the provided modules, and it works alongside Flash Attention, FSDP, and DeepSpeed without rearchitecting an existing pipeline.