MacBERT

Why we included this project

If you work on Chinese-language NLP, MacBERT is a practical upgrade over vanilla BERT without forcing you to change your pipeline. It keeps the same architecture, so you can load it with the usual from_pretrained call and drop it into existing PyTorch or TensorFlow code. The training trick is what sets it apart: instead of masking tokens with [MASK], it replaces them with similar words, which makes the pretraining closer to what the model sees at inference time. That change pays off on Chinese benchmarks, where it outperforms BERT and RoBERTa-wwm on reading comprehension, inference, sentiment, and sentence-pair tasks. Both base and large checkpoints are on Hugging Face, and the project is well documented, so it's a solid starting point if you want a better Chinese encoder that still behaves like BERT.