TextBrewer

Why we included this project

If you need to deploy a BERT-sized model on hardware with limited memory, TextBrewer lets you try knowledge distillation without writing every method yourself. It bundles soft-label training, attention-map losses, hidden-state matching, and multi-teacher setups behind a PyTorch API, so you can compare approaches by changing configuration rather than reimplementing them. The toolkit handles common supervised NLP tasks like text classification, reading comprehension, and sequence labeling, and the example notebooks include a BERT-EMD based custom distiller to help you see how each technique behaves on real checkpoints. The documentation and the ACL paper give you enough background to go deeper when the defaults are not enough.