xlnet

Why we included this project

XLNet's authors released this repository as the reference implementation of their model, which makes it the place to look for how a strong Transformer-based language model is actually pretrained and fine-tuned rather than just described in a paper. It ships pretrained Base and Large checkpoints with the SentencePiece tokenizer and configs, plus working scripts for text classification, regression, and reading-comprehension tasks such as SQuAD. The permutation-based autoregressive objective departed from the masked denoising approach BERT popularized, and the Transformer-XL backbone helps on tasks where long context matters. Since the code targets TensorFlow 1.x and was built around TPU training, it is not a production serving stack today; treat it as a reference for reproducing experiments and understanding the architecture. Teams wanting to build on or learn from a widely studied pretraining method will find that research value worth the setup.