spacy-transformers

Why we included this project

spacy-transformers is Explosion's own bridge between spaCy and Hugging Face's transformers library, so you can keep your existing spaCy pipeline and still benefit from transformer models like BERT, RoBERTa, or XLNet. The hard part it solves is alignment: transformer tokenizers and spaCy's tokenizer do not split text the same way, and this package maps between them automatically instead of leaving that as a manual integration step. It also lets several pipeline components share a single transformer and backprop into it together, which is handy when one encoder serves multiple tasks. Teams that already train with spaCy v3's config files will find it slots into that workflow, and serialization and model packaging are handled out of the box.