semchunk

Why we included this project

Retrieval pipelines and document indexers share an awkward middle step: turning long text into model-ready chunks without slicing through a sentence mid-thought. semchunk handles that split with a deliberately small API. You build a chunker once, then run it over single texts or whole batches, with optional overlap and offsets, and multiprocessing for large corpora. It works with whatever tokenizer you already use, whether that is Tiktoken, Hugging Face Transformers, or a simple custom counter, so it does not lock you into a particular model stack. Teams that want predictable, dependency-light chunking without adopting a heavier framework will find it drops into an existing pipeline quickly, and the optional AI-powered mode adds model-guided boundaries without changing your core setup.