Should You Block AI Crawlers? The Four Switches That Decide If AI Sees Your Site

Aug 29

7 min read

There isn't one AI switch. There are four.

A client rang last month to ask whether he should block ChatGPT from his website. He'd read something somewhere. He wasn't sure what. He just wanted to know which button to press.

There isn't one button. There are four.

They do genuinely different things, and most UK business owners don't know any of them exist. So here's the map. And the short version of whether you should block AI crawlers: almost certainly not.

The short answer

Leave all four switches open. For the overwhelming majority of UK small businesses, blocking AI systems removes you from a place where buyers are already looking and gives you nothing back.

The nuance is worth two minutes, though, because the four controls aren't interchangeable. One decides whether AI companies can crawl you at all. One decides whether Google can use your content for model training. One decides whether you appear in AI-powered search results. One decides whether you get a snippet anywhere, AI or otherwise.

Getting them confused is how a business ends up quietly invisible and can't work out why. Blocking is a decision for people who sell their words. Everyone else is just turning down free distribution.

Switch 1: robots.txt and the AI crawlers themselves

This is the file that tells named bots whether they're allowed in. GPTBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot and others all check it.

On Wix, robots.txt is editable directly in the SEO dashboard, and Wix documents both how to block AI crawlers and how to explicitly allow GPTBot, ClaudeBot and PerplexityBot. It's a five-minute change either way.

What blocking actually costs you: if ChatGPT or Perplexity can't crawl your site, they can't quote you. Not that they'll rank you lower. They won't have you at all. When someone types "commercial electrician in Farnham" into an AI assistant, the shortlist gets built from sites the crawler could read. Yours isn't one of them.

The argument for blocking is that AI companies are training on your content without paying you. That's a fair position for a publisher whose product is the words. It's a very expensive position for a plumbing firm whose product is plumbing. Your service pages aren't intellectual property being stolen. They're a shop window you already put up for free.

Leave robots.txt AI crawlers open unless you're selling access to the content itself.

Switch 2: Google-Extended, the one people confuse with everything else

Google-Extended controls whether Google can use your content to train and improve its Gemini models. That's all it does.

It is not the switch that decides whether you show up in AI Overviews or AI Mode. People conflate the two constantly, usually because both have "AI" in the name and neither has an obvious plain-English label. We covered the underlying problem in more depth in why ranking on Google no longer gets you mentioned by AI.

So the decision here is philosophical rather than commercial. Blocking Google-Extended doesn't protect your traffic and doesn't remove you from AI answers. It just opts your content out of training data. If you feel strongly about that, block it, and understand you've made an ethical choice rather than a marketing one. If you don't feel strongly, leave it alone.

Switch 3: the Search Console "Search generative AI" control

This is the one that actually moves traffic, and UK sites got it first.

On 3 June 2026 the Competition and Markets Authority required Google to give publishers a control over whether their content is used in AI-powered search features, including AI Overviews and AI Mode. The CMA called it a world first. Google has nine months from that date to comply fully. The setting appeared on .co.uk properties from June 2026 and only began rolling out to US and international properties in late July 2026, where the rollout is still partial.

You'll find it in Search Console under Settings, then Search generative AI. It's set at property level, with three states: Include (the default), Exclude, and Inherit from parent. It sits alongside the generative AI performance reports we walked through in what UK sites can finally see in Search Console.

What Exclude does:

Your content stops being visible in, linked from, or used to ground responses in AI Overviews, AI Mode and generative AI in Discover.
You get no traffic and no impressions from those surfaces.
It takes effect within one to two days.

What Exclude does not do:

It isn't a ranking signal. It won't move you up or down in standard search results.
It doesn't control AI model training. That's Google-Extended's job.
It doesn't stop your content surfacing in the Gemini app.

There's a side effect worth knowing about. On 17 July 2026 Google confirmed the Top Stories carousel is fully live inside AI Overviews, currently US mobile only. The direction of travel is clear enough: opting out of AI search increasingly means opting out of the features built on top of it.

This Search generative AI control is the most consequential of the four. It's also the one where Exclude sounds safest and is usually the most damaging thing you can pick.

Switch 4: nosnippet at page level on Wix

nosnippet is the bluntest instrument here, and it's the one people reach for when they want something more surgical than a site-wide setting.

On Wix you set it per page: Advanced SEO, then the Robots meta tag section, tick nosnippet, publish. Done in under a minute.

Here's the problem. AI Overviews can only cite pages that are eligible for snippets. So nosnippet does remove your AI Overview eligibility, but it also removes your normal search snippet. That grey descriptive text under your blue link in Google results disappears. You've swapped a targeted AI opt-out for a worse listing in ordinary search.

If a page is sensitive enough that you don't want it quoted, the right answer is usually to noindex it or not publish it. Not to strip its snippet and hope. nosnippet doesn't fine-tune anything. It removes you from the snippet, and the snippet is how you get cited.

A decision rule you can actually use

Find your situation, then do the thing next to it:

UK SME, trades, professional services or local business — leave all four open. Include in Search Console.
Service business relying on enquiry forms and calls — leave open. Being absent doesn't send the enquiry somewhere better.
You publish original research or data others quote — leave open. Add clear authorship, dates and sources so citations point back to you.
Subscription or paywalled publisher — blocking is a real option. Consider robots.txt and the Search Console exclusion together, not separately.
You host licensed or third-party data you don't own redistribution rights to — check the licence first. That's a legal question, not an SEO one.
One page contains sensitive or confidential detail — don't use nosnippet. Noindex it, gate it, or don't publish it.

Page-level AI controls are expected under the CMA compliance deadline, but nothing has been announced. Plan around what exists today.

What to do with the five minutes instead

Stop worrying about being seen and start worrying about being worth quoting.

The first randomised field experiment on AI Overviews came out this year. Saharsh Agarwal at the Indian School of Business and Ananya Sen at Carnegie Mellon's Heinz College ran 1,065 desktop Chrome users, collected data between 7 January and 10 February 2026, and posted the paper on 3 April 2026. When an AI Overview appeared, outbound organic clicks fell 39.8% and zero-click searches rose 34.5%. User satisfaction didn't improve. Sponsored clicks were unaffected.

Read quickly, that's an argument for blocking. Read properly, it's the opposite. If four in ten clicks are evaporating, the citation itself becomes the asset. Being named in the answer is now a large part of what showing up means. Removing yourself from the answer doesn't win the click back. It hands it to whoever's left.

Being citable is unglamorous work. Clear headings that match real questions. Direct answers in the first two sentences of a section, before the elaboration. Named authors with actual credentials. Dates on everything. Prices, service areas and specifics stated plainly rather than buried in a hero image. Structured data that matches what's on the page. On Wix Studio all of that is available without a developer, which is more than can be said for most platforms. We showed exactly how it plays out in the case study of a local gas engineer who ended up inside Google's AI answer.

The bit where I'll be blunt

I've had this conversation a dozen times in the last few months, and it follows the same shape. A business owner reads a headline about AI eating search traffic, feels a reasonable urge to protect something, and goes looking for a switch to flip. The switch feels like control. It isn't. It's a way of converting anxiety into an action, which is not the same as converting anxiety into a result.

The businesses I see winning in AI answers right now aren't doing anything exotic. They've got well-structured pages, clear service descriptions, real authorship, and consistent information across their site, their Google Business Profile and their directory listings. That's it. AI systems reward the same clarity human buyers do, which shouldn't be a surprise given what they were trained on.

The question isn't whether AI can see your site. It's whether it has anything worth quoting when it does.

Matt Konarzewski, Vision Marketing

If you're not sure which of these four is switched on across your own site, it's a ten-minute check and worth doing properly. Get in touch with Vision Marketing and we'll go through it with you.

Questions clients actually ask

Does blocking AI crawlers protect my content from being copied?

No. Blocking a well-behaved crawler like GPTBot or ClaudeBot stops those specific systems from reading your site. It does nothing about people copying and pasting, scrapers that ignore robots.txt, or content that's already been crawled.

If I set Exclude in Search Console, will my Google rankings drop?

No. Google states plainly that the Search generative AI setting is not a ranking signal and doesn't affect your position in standard search results. You'll simply stop appearing in AI Overviews, AI Mode and generative AI in Discover.

What's the difference between Google-Extended and the Search Console AI setting?

Google-Extended controls whether your content is used to train Google's AI models. The Search Console control decides whether your content appears in AI-powered search features. Blocking one doesn't affect the other.

How do I change AI crawler settings on Wix?

For site-wide crawler rules, edit robots.txt in the Wix SEO dashboard. For a single page, use Advanced SEO and the Robots meta tag. The Search generative AI setting lives in Google Search Console rather than Wix, because it's a Google control, not a platform one.

Is nosnippet ever the right choice?

Rarely. It removes AI Overview eligibility, but it also removes your ordinary search snippet, which usually costs more than it saves. If a page shouldn't be quoted at all, noindex is the cleaner tool.