gigatoken

Why we included this project

Tokenization rarely gets attention until it is the slow step in your training loop. Gigatoken is a Rust-based tokenizer that turns text into token IDs for LLM training and fine-tuning, and its compatibility wrappers let you keep using HuggingFace Tokenizers or tiktoken while getting faster throughput without changing your data code. The native API is where the multi-GB/s numbers come from, since the Rust implementation reads text files directly and parallelizes aggressively. Benchmarks cover popular models on server and laptop CPUs, so the speedup figures are grounded in real hardware. Teams that tokenize large corpora repeatedly, whether in dataset preparation, pretraining, or distillation, are the ones most likely to save serious wall-clock time.