matmulfreellm

Why we included this project

Anyone poking at efficient LLM architectures will find this repo useful. It implements MatMul-Free LM, a design that removes the matrix multiplications at the heart of standard transformers, replacing most linear layers with ternary weights and gated recurrent attention. That trade is the interesting part: you shed a lot of FLOPs and memory traffic while keeping generative quality close to a conventional model. The repo ships pretrained checkpoints at 370M, 1.3B, and 2.7B that load through the standard Hugging Face AutoModel interfaces, so you can generate text or fine-tune without much setup. Because the code is adapted from flash-linear-attention, anyone who already knows that library can see how ternary quantization.