BitNet

Why we included this project

BitNet pushes model quantization about as far as it can go by storing weights at roughly 1-bit precision instead of full-precision floats. This repo is a PyTorch port of that design: a BitLinear layer you can drop into a standard transformer wherever you'd normally use nn.Linear, plus example scripts, a multi-grouped-query attention variant, and a full BitNetTransformer class ready to instantiate. The README is upfront about the catch: a model has to be trained or fine-tuned from scratch for these layers to help, so you can't just retrofit an already-trained checkpoint. If you want to see how low-bit language models behave in practice, this is a solid place to start. Just expect rough edges, since parts of the codebase are still being reworked.