minbpe

Why we included this project

minbpe offers a small, readable implementation of byte-level Byte Pair Encoding, the tokenization method behind GPT, Llama, and Mistral. It includes three tokenizer classes: a basic one, a regex-based variant that prevents merges across category boundaries, and a GPT-4 wrapper that matches tiktoken's output exactly. All three handle training, encoding, and decoding, which makes the code a practical starting point for developers who want to build a custom tokenizer or understand how tokenization works before reaching for tiktoken's optimized internals. Because the code is deliberately minimal, it also works well as a teaching tool for teams getting familiar with LLM fundamentals. Teams that need predictable, auditable tokenization can adapt it as a reference implementation.