#5 · Primary category： Privacy-Preserving & Federated Data Science
ai.robots.txt
A list of AI agents and robots to block.
Project last updated：08/26/26
GitHub Stars
4.1K
Forks
179
Contributors
75
License
MIT
Why we included this project
If your site is getting crawled by AI agents you never invited, this list gives you a quick way to push back. It tracks AI-related bots in a community-maintained registry and ships blocking rules in several formats: a standard robots.txt plus ready-made snippets for nginx, Apache, Caddy, HAProxy, and lighttpd, so you can paste in the one that matches your server instead of hand-writing rules. The accompanying bot-metrics table documents what each crawler is for, which makes the project handy as a reference when you want to know who is crawling your site. It is configuration.
Articles for this project
No articles for this project yet.
To suggest a topic or contribute an article, contact us.
Related projects in this category
watermarks-remover
A privacy-first app that strips AI watermarks from content you own.
PySyft
Perform data science on data that remains in someone else's server
presidio
An open-source framework for detecting, redacting, masking, and anonymizing sensitive data (PII) across text, images, and structured data. Supports NLP, pattern matching, and customizable pipelines.
flower
Flower: A Friendly Federated AI Framework
deep-prove
Framework to prove inference of ML models blazingly fast