ai-crawler-py

Why we included this project

Most web scraping tools force you to hand-write CSS selectors or XPath rules before you can pull anything. This Python package takes a different route: you give it a starting URL, describe the data you want in plain English like 'find all Halo games and their prices,' and it explores the domain, picks out the relevant pages, and returns Markdown or JSON shaped by a schema you also describe in words. The crawl engine itself runs on Oxylabs' hosted AI Studio service, so JavaScript-heavy pages and geo-targeted requests are handled on their side rather than on your own machine. That is what makes it practical for teams that need pricing or product data for analysis without standing up and maintaining scraper infrastructure. The schema helper is a nice extra, turning a loose prompt into structured output that slots into existing pipelines.