spider

Why we included this project

Spider is a Rust crawling engine built around concurrency. It streams pages back the moment they're fetched and only fires up headless Chrome for pages that actually need JavaScript rendering, so most of a typical crawl never touches a browser. The engine scales from a single script to a distributed setup without you changing your code. It ships as a Rust library plus a CLI, with Node and Python packages, and an optional managed cloud mode takes over proxy rotation and anti-bot handling once local prototyping stops scaling. You can export what it collects as Markdown, JSON, or WARC, which drops cleanly onto vector-store ingestion and site-monitoring workloads. Teams that need a steady feed of fresh web content for LLM or RAG pipelines will find it a solid, production-oriented starting point.