nlp-lang

Why we included this project

This Java library collects the kind of utility code that keeps reappearing in Chinese-language NLP projects, so you don't have to rewrite it each time. It covers word normalization, sentence splitting, HTML tag stripping, simplified-traditional conversion, and trie structures for fast lookups, then adds extras like pinyin conversion, Bloom filters, SimHash-based duplicate detection, in-memory search suggestions, and word frequency/IDF statistics. The double-array trie and Viterbi support are there too, which matters if you're building a tokenizer or search index. It's not a full pipeline, just a grab bag of primitives you assemble yourself, but that's exactly what teams writing custom text-processing layers usually need.