crawlee

Why we included this project

Teams that need fresh, structured web data to feed an LLM, RAG index, or fine-tuning set will find Crawlee a practical choice. It's a Node.js library that handles the whole crawling job: it keeps a persistent URL queue, rotates proxies, and generates browser fingerprints that make scrapers look human, so bot protection is less likely to block them. You can scrape fast over raw HTTP with Cheerio or JSDOM, or switch to a real headless browser via Playwright or Puppeteer when a page needs JavaScript rendering, and the same interface works for both. Results go to pluggable storage on disk or in the cloud. The built-in retries, session management, and scaling save teams the fiddly work of keeping scrapers reliable.