Does robots.txt Actually Stop AI Scrapers?

The most common advice for blocking AI bots is to edit your robots.txt. It helps, but it is not the full answer, and trusting it alone leaves you exposed. Here is the honest version of what robots.txt can and cannot do.

What robots.txt actually is

robots.txt is a plain text file that politely asks crawlers not to visit certain pages. It is a request, not a lock. A bot has to choose to read it and choose to obey it. Nothing on your server enforces it.

Which bots obey it, and which do not

Well-behaved crawlers such as Googlebot, Bingbot, and OpenAI's GPTBot generally honor robots.txt. Plenty of others, including some of the highest-volume scrapers, ignore it entirely, or read it only to learn which pages are worth taking.

The catch nobody mentions

Your robots.txt is public. Anyone, and any bot, can read it at yoursite.com/robots.txt. So a Disallow line also doubles as a map of exactly what you did not want seen, which is not ideal against bad actors.

What actually stops scraping

To truly stop a crawler you block it at the server or edge, by the signature of the request itself, so it never reaches your pages whether or not it read your robots.txt. You can run a free check with the free StopBotAPI scanner to see which bots are getting through yours right now, then block them in one line with StopBotAPI.