trafilatura

Why we included this project

Anyone who has tried feeding scraped web pages into a language model or a search index has met the same problem: most of the page is chrome. Cookie banners, navigation menus, and footer links bury whatever signal you were after, and Trafilatura exists to cut through that. It isolates the main article body, metadata, and comments and returns them as clean structured output in plain text, Markdown, JSON, or XML-TEI, working equally well as a Python library or a command-line tool. Teams building training corpora or RAG pipelines, and anyone running news aggregation, use it to normalize raw HTML into dependable text before anything else in the stack touches it. If fetching many pages and turning them into noise-free content is a regular part of your work, this quietly saves hours of cleanup per project.