How to Block Bytespider (ByteDance's Crawler)
Bytespider is the web crawler run by ByteDance, the company behind TikTok. It is one of the highest-volume AI crawlers on the internet, and it has a reputation for ignoring robots.txt and hammering small sites. Here is how to find it in your logs and stop it.
How to spot Bytespider
Look in your access logs for the user-agent Bytespider, often arriving in bursts of rapid requests from many addresses at once. If it is on your site, the free StopBotAPI scanner will flag it along with the other AI crawlers hitting you.
Why robots.txt usually will not stop it
Unlike crawlers that publish a policy and mostly honor robots.txt, Bytespider is widely reported to ignore it. Adding a Disallow line is worth doing, but you should assume it will keep coming, which means you need blocking that does not depend on the bot cooperating.
How to actually block Bytespider
Effective blocking checks each request's signature and turns away known training crawlers like Bytespider before they load a single page, while letting search engines and real visitors straight through. StopBotAPI does this with one line you paste into your site and updates the signatures as the bots change.
See which AI bots are on your site
Run a free scan with no signup, then block them in one line.