gse

Why we included this project

Gse brings the jieba segmentation approach to Go, which is useful when Chinese, Japanese, and English text share one codebase and you want dictionary-driven tokenization without running a Python service on the side. It provides several segmentation modes, including precise, common, search-engine, full, and HMM, and the Viterbi machinery handles words that never appear in the dictionary, a familiar problem with CJK input. Companion adapters connect it to Elasticsearch and Bleve, so teams already indexing multilingual content can keep tokenization close to their data. Because it is a library.