trankit

Why we included this project

Trankit is a transformer-based NLP toolkit that handles the usual text-processing chores across many languages: sentence segmentation, tokenization, part-of-speech and morphological tagging, dependency parsing, and named entity recognition. Its pretrained pipelines cover 56 languages and are built on XLM-RoBERTa, and the multilingual mode can detect the input language automatically, so you can throw mixed-language documents at it without picking a language first. The package is light enough to run on a laptop, and the Python API plus a command-line interface make it easy to script for research or embed in an application. You can also fine-tune the pipelines on your own data instead of being stuck with the pretrained models.