crawlee-python

Why we included this project

Crawlee is a Python library for web scraping and browser automation that handles the unglamorous plumbing of crawling, from request queuing and retries to session management and proxy rotation. That frees you to write the extraction logic instead of the machinery around it. You can grab HTML, PDFs, images, and other files for LLM, RAG, or analytics pipelines, and pick between lightweight parsers like Parsel and BeautifulSoup or full browser automation through Playwright, in headful or headless mode. Results land in machine-readable storage backends, so wiring scraped data into downstream processing is simple. If your crawler needs to look human and slip past modern bot protections, the defaults already get you most of the way there, and the configuration options cover the rest.