news-please

Why we included this project

news-please saves you the tedium of scraping news sites yourself. Point it at a root URL and it follows internal links and RSS feeds to pull out the headline, body text, lead paragraph, authors, publication date, and language, then dumps the result into JSON, Elasticsearch, PostgreSQL, Redis, or wherever you want it. If you only need article extraction inside an existing pipeline, the library mode gives you that without spinning up a full crawl; the CLI is there for whole-site runs and can revisit articles to track revisions over time. For research teams assembling historical news corpora, the Common Crawl integration stands out, since it lets you filter that public archive by outlet and date instead of scraping each publisher one by one. The stack underneath is battle-tested Scrapy, Newspaper, and readability, so you get dependable extraction rather than experimental parsing logic.