Blocking or allowing “AI bots” is no longer a one-click decision. Behind that label there are at least four types of visitor with different goals, and the rule actually enforced on your website is not always the one your marketing team thinks it is. If nobody in senior management owns this decision, your company may be disappearing from ChatGPT or Gemini answers without knowing it, or giving content away for nothing in return.
What has happened: the debate is no longer “yes or no”
An idea is gaining ground in the international SEO community that is worth taking to the management team: companies need to move from improvised block lists to a documented, agreed bot policy. The argument starts from a simple distinction. What we usually call an “AI bot” is in fact several very different things:
- Training crawlers, which read the web to train language models.
- Retrieval bots, which fetch pages in real time to build an answer in an AI search engine or assistant.
- User-initiated visits, when someone asks an assistant to read a specific page.
- Autonomous agents, which browse, compare and even try to buy as a person would.
On top of these come traditional search engine crawlers. Blocking one or another has very different consequences: shutting out a training crawler protects your content; shutting out a retrieval bot can remove your brand from AI answers altogether, not just push it down a few positions.
The problem almost nobody sees: robots.txt is not in charge
The robots.txt file, where most websites declare which bots they accept, is a voluntary request. What actually blocks traffic is the network and security layer: the CDN, the web application firewall or the bot management service. And that layer is usually configured by another team, with other goals (stopping abuse, saving server capacity, protecting data).
The outcome is common: marketing believes the site is open to AI assistants because robots.txt says so, while the firewall is blocking them by default. Or the reverse: the company thinks it is protecting its catalogue and prices, and is in fact serving them without control. On top of that, agents that browse like people trigger analytics and blend into real traffic, distorting conversion rates and acquisition costs.
Why it matters to your business
Visibility in AI is starting to matter for customer acquisition. When a buyer asks an assistant about suppliers, products or comparisons, only the websites the assistant has been able to read show up. A misconfigured CDN rule can silently cut off that channel, and nobody will spot it in the monthly report because there is no visible drop: you are simply not there.
There is also a question of margin and risk. Opening everything has costs: server load, your own content feeding third-party models with no return and, in B2B catalogues, prices or terms that should not circulate. Closing everything has others: less presence where customers are already searching. It is a business decision with effects on sales, costs and exposure, and that is why it should not be left to a single technical person.
What to check this week
You do not need a big project to know where you stand. These questions, which any managing or marketing director can ask, are usually enough to spot the problem:
- Do the rules match? Compare what robots.txt says with what the CDN and firewall actually block. If nobody can answer, that is your first task.
- Who is really coming in? Ask your technical team to review the server logs for recent weeks and identify which AI bots arrive, which pages they visit and how often.
- What do you want to protect, and what do you want to be seen? It usually makes sense for AI to read product pages, categories and brand content, but not private areas, customer-specific prices or internal search.
- Who decides? Marketing, technology, security, legal and sales should agree the policy and review it every few months, because bots change fast.
An illustrative example: a B2B distributor may decide to let AI assistants read its public catalogue so it appears in comparisons, while blocking automated access to the customer area and price lists, and separating agent traffic in analytics so it does not inflate conversion metrics.
How we approach it at TRILOGI
The first thing we do is put the facts on the table: we cross-check robots.txt, the CDN and firewall configuration and the actual server logs to see which AI bots get in, which are kept out without anyone having decided it, and which pages they read. With that map, we propose a bot policy by type (training, retrieval, user-initiated and agents) aligned with commercial goals, apply it at the layer that actually enforces it and set up a regular review so it does not drift with every update. It is part of the work we do on our managed eCommerce platform, where security and visibility are configured together rather than against each other.
The question for your next management meeting is simple: who in the company decides which artificial intelligences can read our website? If the answer is “nobody” or “the hosting provider”, it is worth an hour of your time.
Every Tuesday, the 5 news stories that matter: subscribe to the newsletter.




