#82 · Primary category： Knowledge Base & RAG
webclaw
Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.
Project last updated：08/26/26
GitHub Stars
2.3K
Forks
232
Contributors
9
License
AGPL-3.0
Why we included this project
Extracting usable content from the web is often the messiest step in building an LLM app, and webclaw is aimed squarely at that problem. Written in Rust and self-hostable, it converts pages into clean markdown or JSON a model can actually consume, rather than dumping raw HTML full of navigation and scripts on your agent. You can drive it from a CLI, a REST API, or an MCP server, which covers both live agent workflows and scheduled ingestion into a knowledge base. The crawler also gets past bot protection, renders JavaScript, and handles TLS fingerprinting, the things that usually trip up simpler scrapers, and a local-first mode means you can run it without an account. If your pipeline depends on fresh or structured web data, it is a credible open-source replacement for hosted extraction services that charge per page.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ragflow
RAGFlow is a leading open-source Retrieval-Augmented Generation (RAG) engine that fuses cutting-edge RAG with Agent capabilities to create a superior context layer for LLMs
Understand-Anything
Graphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
crawl4ai
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN
docling
Get your documents ready for gen AI
anything-llm
Stop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experience