anakin

Why we included this project

Teams wiring LLM applications to live websites will find a lot to like here. A single HTTP request turns any URL into clean markdown, or structured JSON when you supply a Gemini key, so it feeds straight into RAG pipelines, agent tooling, and dataset collection without a preprocessing step. The handler chain starts with a free local HTTP fetch and only escalates to an anti-detect Firefox browser or a paid third-party API when a page resists, which keeps most scraping off the cloud tab. Proxy selection uses Thompson Sampling to learn which rotating address works best for each domain instead of cycling through a fixed rotation. Since it ships as a single Go binary with a make up Docker stack and a zero-config in-memory mode, you can run it on your own hardware and avoid per-scrape SaaS pricing.