AI Labyrinth: Cloudflare Defense Against Rogue AI Crawlers
AI Labyrinth, a novel mitigation technique that slows down, perplexes, and wastes the resources of AI Crawlers and other bots who disregard “no crawl” instructions by using AI-generated content
Four of the top 20 Facebook posts from last autumn were purportedly AI-generated content, demonstrating the explosion of this type of content
Every day, more than 50 billion requests or slightly less than 1% of all web requests are sent to the Cloudflare network by AI crawlers
AI Labyrinth also serves as a next-generation honeypot, which is an extra bonus. A true person wouldn’t navigate through a labyrinth of AI-generated gibberish
Cloudflare combined Workers AI with an open source model to produce original HTML pages on a variety of subjects in order to produce material that is convincingly human-like
Cloudflare discovered that more varied and compelling outcomes were obtained when a wide selection of themes was first generated, followed by content creation for each topic
Cloudflare made sure that only suspected AI scrapers are shown these URLs, allowing confirmed crawlers and legitimate users to browse regularly, in order to further reduce the impact on ordinary visitors
Cloudflare can find new bot patterns and signatures that could otherwise go unnoticed by examining which crawlers are using these covert routes
The AI Labyrinth requires no further configuration once it is enabled; it starts operating right away