#3 · Primary category： Knowledge Base & RAG
crawl4ai
🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN
Project last updated：08/29/26
GitHub Stars
79.9K
Forks
8.3K
Contributors
91
License
Apache-2.0
Why we included this project
Most scrapers hand you raw HTML and leave the cleanup to you. Crawl4AI does that work itself: it renders JavaScript-heavy pages in a headless browser and returns clean Markdown with navigation and boilerplate already stripped out. That output is what makes it useful for retrieval sources, agent workflows, and fine-tuning datasets, since you can feed it to a model without a preprocessing stage. The Python API is simple enough to drop into an existing pipeline quickly, and the project is under active development with a large community behind it.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
ragflow
RAGFlow is a leading open-source Retrieval-Augmented Generation (RAG) engine that fuses cutting-edge RAG with Agent capabilities to create a superior context layer for LLMs
Understand-Anything
Graphs that teach > graphs that impress. Turn any code into an interactive knowledge graph you can explore, search, and ask questions about. Works with Claude Code, Codex, Cursor, Copilot, Gemini CLI, and more.
docling
Get your documents ready for gen AI
anything-llm
Stop renting your intelligence. Own it with AnythingLLM. Everything you need for a powerful local-first agent experience
meilisearch
A lightning-fast search engine API bringing AI-powered hybrid search to your sites and applications.