If you’ve published content online recently, there’s a good chance GPTBot (OpenAI’s web crawler) has already collected it. You may think: “I don’t want AI to steal my content!” But, maybe you do? Whether or not to block AI crawlers is a topic anyone with a website needs to think about these days.
Cloudflare recently made it possible to block AI bots with one click in their dashboard. According to the hosting platform, there has been an explosion of AI bot activity. “AI Crawlers generate more than 50 billion requests to the Cloudflare network every day,” the company wrote in March. All our clients’ websites are on Cloudflare, so we did some digging into the pros and cons of blocking AI crawlers.
What blocking AI crawlers means in practice
If you allow AI crawlers, your site’s content can be pulled into AI tools like ChatGPT, Perplexity, or Google’s AI Overviews. If you block them, your content won’t be used to train models or appear in AI-driven answers.
Big media outlets are already blocking AI, so should you do the same?
Here’s the short answer: probably not.
Why some businesses are blocking AI crawlers
The sites making headlines for blocking AI are mostly media companies (including ADWEEK, BuzzFeed, Inc., Fortune, News/Media Alliance, The Atlantic, Battelle Media, Evolve Media, O’Reilly Media, Third Door Media, TIME) that rely on page views and ad revenue. If AI tools serve up their articles without sending readers to their site, that’s lost income, so blocking crawlers makes sense for them.
But most of our clients (small, service-based businesses) are not selling ad space. They’re trying to build visibility, trust, and leads.
What blocking means for visibility
When you block AI crawlers, you’re basically saying: “Don’t use my content in AI results.” That means fewer chances for your brand to appear in tools people are actually using. For example, I asked ChatGPT to “name the top boutique CPA firms in Jacksonville” and one of our client firms showed up. But if their site had been blocked, they wouldn’t have been listed.
If your marketing goal is awareness (and for most small firms, it is), why would you want to opt out of the next big discovery channel?
But what about data theft?
Yes, it feels strange to realize that AI companies scrape your site to train their models. But if it’s on the open web, it’s already public. Anyone can read it, quote it, or share it; AI is just doing it at scale.
Understandably, that might make you feel uncomfortable. You’ve put time and resources into creating something for your site, and it could be aggregated into an AI-powered search result without any attribution.
But we have to be realistic about how people are finding information these days. In my opinion, for many companies, the bigger risk isn’t AI stealing your content; it’s making yourself invisible in the places people look for answers.
The good news is that Google’s AI Overviews link to sources. ChatGPT doesn’t always link, but if your site is known, it can still surface your brand. Either way, if you block crawlers, you’re shutting the door on visibility. Here’s why:
- AI Overviews and featured snippets together can take up to 75.7% of the screen on mobile
- ChatGPT prompt volume jumped nearly 70% from January to June 2025
- Shopping queries doubled in popularity over six months
- Users are clicking links in ChatGPT at more than twice the rate they were just a few months ago
Our take on blocking AI crawlers
For most small businesses and professional services firms, we recommend keeping your site crawlable. It’s not worth walling off content when your goal is to be found. However, there are some exceptions, such as if you’re in a highly regulated space where data use could be risky, or if you have private, member-only, or gated content you truly don’t want scraped. Otherwise, let the bots in so your business can join the AI-driven future of search.
Where all this is headed
Right now, Cloudflare’s toggle is a bit of a blunt instrument. Although you can choose to only block AI crawlers on hostnames with ads, you can’t say yes to Google AI Overviews but no to ChatGPT.
That’s why most small businesses are better off leaving crawlers on for now. The real solution (coming soon, no doubt) will be finer controls. Ideally, you could allow some bots (like Google’s AI Overview crawler that links back) and block others. For now, it’s all-or-nothing. Until better options exist, we lean toward letting AI crawlers do their thing.
And, while you’re at it, give your site the best chance of showing up in AI searches by sticking to the search fundamentals: a fast, user-friendly website, high-quality content that matches user intent, and trust signals like reviews and online brand mentions. If you need help creating an online presence that’s more likely to show up, give us a shout!
