The Day We Found Out Every AI on the Internet Couldn't Read Our Website

Yesterday morning I was testing something unrelated and pointed an AI model at our own website. The model came back with a 403 error. I tried a different one. Same result. Tried a third. Same.

Every major language model on the internet — ChatGPT, Claude, Perplexity, Google’s Gemini — could not read v8gp.co.uk. For any user asking an AI about V8 Global, Axia, or anything we’ve written, the model would either say it couldn’t find information, or worse, hallucinate something.

The site had been live for two weeks. We had no idea.

The problem nobody would have flagged

Here’s what makes this uncomfortable. Google still indexed us. Human visitors still loaded the site. Every monitoring tool showed green. The only thing broken was the thing we didn’t think to measure — whether AI agents, the channel that’s becoming how a chunk of our prospects discover us, could actually read our content.

The cause: Cloudflare had a default setting called “Block AI Bots,” enabled automatically since mid-2024. The intent is reasonable — protect your content from being scraped for AI training without your consent. The side effect is that the same filter also blocks the user-facing AI fetchers. When a prospect asks ChatGPT “what does V8 Global do?”, ChatGPT’s browser tries to read v8gp.co.uk and gets a 403. It’s a default I suspect thousands of businesses have on without knowing it.

Fifteen minutes to diagnose. Five minutes to fix. Two weeks we’d been invisible to the channel we’re positioned to win.

Then we looked at the security headers

While the AI bots situation was being resolved, I ran the site through Google’s standard audits. PageSpeed Insights on a blog post came back with 91 for Performance and SEO, 93 for Accessibility on mobile — all respectable. Then there was Best Practices: 77.

A 77 on Best Practices sounds fine. It isn’t. What it actually meant: three security headers missing, each flagged as “high severity” by Google’s auditor.

Content-Security-Policy. This is the header that tells a browser “only load scripts from these specific domains.” Without it, if an attacker ever managed to inject a malicious script into your page — through a form, a comment system, a compromised third-party widget — the browser would run it. With a proper CSP, the browser refuses.

Strict-Transport-Security. This forces browsers to only ever connect over HTTPS, even if a user typed the non-secure version of the URL.

Cross-Origin-Opener-Policy. This isolates your site’s browser window from other sites’ windows, preventing a class of attacks where a malicious popup could read information from your page.

None of these had been configured. Our Cloudflare deployment shipped them as “not set,” and the default was to leave that door unlocked.

The fix took forty minutes. The hardest part was writing the CSP, because a CSP is only useful if it’s strict — and strict means you have to explicitly list every third-party service your site calls out to. Google Fonts. YouTube. Google Analytics. Google Maps. Miss one and that thing breaks. Leave one out and the CSP is theatre.

What this day cost, and what it would have cost

I counted. Between diagnosing the AI block, writing and testing the CSP, fixing the accessibility issues the audit surfaced, and verifying everything didn’t break in the process — the whole pass took roughly four hours spread across the day.

The other option, the one most SMEs take, is to not do any of this. The site stays live. It looks fine. Nobody emails to complain. Six months later, when the business starts wondering why the content marketing investment hasn’t generated the AI-search visibility everyone’s been promising, nobody traces it back to a Cloudflare toggle that was on since day one.

Four hours is what discipline costs. The absence of discipline compounds quietly and bills you later.

Why this is the real job

When people look at a website they evaluate it on the surface. Design quality, copy, how the homepage hero makes them feel. All of that matters. None of that is the work.

The work is the layer nobody sees. Is the site readable by the channels you depend on. Is the security posture adequate for the data you’re collecting. Does the canonical URL tell search engines the right thing. Does the CSP allow the embed you shipped last month and also deny the attack vector you haven’t thought of. Does the form backend write what it claims it writes. Does the export endpoint still work. Does the new blog post actually appear in the RSS feed.

An SME operator who cannot answer those questions about their own website doesn’t have a broken site. They have a site that works today and will betray them in some way they won’t be able to diagnose six months from now.

This is the gap our clients keep running into when they come to us. Not “we need a new website.” The website is fine. The operator layer underneath it — the discipline to find problems before they surface as symptoms — is what’s missing.

The list I carry

If you run an SME site and want to do this honestly, here’s the short list:

• Can AI user-agents fetch your pages? Test GPTBot, ClaudeBot, Perplexity. If any return 403, you have the same problem we had.
• Does your site have Content-Security-Policy, Strict-Transport-Security, and Cross-Origin-Opener-Policy headers set? Run it through Google PageSpeed Insights and look at the Best Practices tab.
• Does your robots.txt make your intent explicit, including for AI crawlers?
• Is there a llms.txt at the root, and do your response headers point to it?
• Does your structured data validate in Google’s Rich Results Test?
• Does your form backend have a working export path, and is it behind something better than a weak hardcoded key?

If you can’t answer all of those in five minutes, you have operator debt. It’s not urgent. Nothing is on fire. But it’s the kind of debt that gets paid back at the worst moment, by the person least equipped to fix it.

The day in one sentence

The website is live. The website is fine. The website is also fundamentally different today than it was yesterday, and nothing on the surface reflects that. That’s the job.

---

Alan Law is founder of V8 Global and architect of Axia. Operator’s Log posts document how AI-native systems get built — and operated — in practice. For the community side of V8, follow Gina Cheng.