tokenizers

Why we included this project

Most people training transformer models have already used this library without knowing it, because it powers the tokenization layer under the Hugging Face Transformers stack. It implements Byte-Pair Encoding, WordPiece, and Unigram, and lets you train a custom vocabulary on your own corpus just by pointing it at a folder of text files. The Rust core makes both training and tokenization fast, to the point that a gigabyte of text on a server CPU takes under twenty seconds, so preprocessing does not stall even at scale. It also takes care of the fiddly bits of input preparation, truncation, padding, and adding special tokens, while tracking alignment so you can always map a token back to the exact span of the original sentence. Teams that need tokenization to match exactly between training and serving, or that want bindings in Python, Node.js, or Rust, will find a dependable foundation rather than a throwaway utility.