Kiwi

Why we included this project

Korean text processing teams often find that tokenizers either run too slowly or fall apart on ambiguous sentences. Kiwi is a C++ morphological analyzer that splits Korean into morphemes using the Sejong part-of-speech tag set, and it includes a small language model to pick the right analysis when a sentence could go several ways. It also corrects simple typos, splits sentences, and uses multiple threads so large batches of text don't crawl. The core is wrapped for several languages, including Python, so you can use it without rebuilding your pipeline. If you need Korean word segmentation and POS tagging that holds up at scale, it's worth running against your own data.