mcp-crawl4ai-rag

Why we included this project

This MCP server gives you a working loop for letting AI coding assistants answer questions from live web documentation: it uses Crawl4AI to scrape a page or an entire site, stores the cleaned, chunked text in Supabase, and exposes a semantic search tool that feeds RAG queries. The included tools cover everything from a single crawl to recursive site indexing with smart URL detection, and you can filter retrieved sources by domain to keep answers focused. Several optional retrieval upgrades are worth trying, including hybrid vector and keyword search, cross-encoder reranking, and a Neo4j knowledge graph that scans a GitHub repo to flag hallucinated imports and method calls in generated Python. One honest caveat: the author calls this a testbed rather than a finished product, so expect a young codebase that runs on Docker and needs your own Supabase project and OpenAI embeddings. If you are wiring an assistant to documentation that changes often, this saves you from assembling the crawling and retrieval pieces yourself.