lazynlp

Why we included this project

Most people building language models discover that the hard part isn't the model; it's getting a clean, large enough corpus to train on. lazynlp attacks that problem directly: it gives you a simple pipeline that crawls web pages, strips them into plain text, and removes duplicates, so what's left is a genuinely usable monolingual dataset. It includes helpers to fetch URL lists from sources like Reddit dumps and Project Gutenberg, along with utilities for boilerplate removal and content cleanup. That makes it a sensible starting point for researchers and engineers who need to assemble training data without writing their own scraping and cleaning code from scratch.