gpt-crawler

Why we included this project

Assembling the training material is usually the tedious part of building a custom GPT or OpenAI assistant that answers questions about your own product or documentation, and this tool automates that step. You point it at a starting URL, add a link-matching pattern and a CSS selector for the content you care about, and it crawls the site, extracts the relevant text, and writes a single output file sized and tokenized for direct upload to OpenAI. It also handles practical details like page limits, skipping binary resources, and splitting large outputs, and it runs as a CLI, a Docker container, or an Express API with Swagger docs. That makes it useful for teams who want a support or documentation assistant grounded in their own content without hand-curating every page. The Builder.io team built it to feed their own docs into a custom GPT, and the same workflow transfers to any site you control.