chonkie

Why we included this project

If you have built a retrieval pipeline by hand, you know the pain of writing text splitters that behave oddly at paragraph boundaries. Chonkie packages the common chunking strategies, from simple token and sentence splits up to semantic and LLM-driven ones, into one lightweight Python library, so you can swap approaches without maintaining splitter code yourself. It can also run the whole ingestion path, fetching a document, cleaning it, chunking, refining, embedding, and handing the result to a vector store, and an optional self-hosted REST API exposes the same logic to non-Python services. The default install stays small and integrations cover dozens of tokenizers, embedding providers, and vector databases, which keeps splitting consistent and reproducible across different retrieval setups.