← Articles

26 trainers ChatGPT cannot see. What an audit of one site says about “ranking in AI”

20 min read

State of knowledge: November 2026.

A training company’s website lists 26 trainers, with photos and descriptions. The crawlers of ChatGPT, Claude and Perplexity get that page without a single name.

The company asked me how much it costs to rank a site in ChatGPT, Gemini and Perplexity. Before I answered, I checked what a crawler gets from that site, not what a person sees. I found five things. Then I checked in published research which of them are backed by data and which merely sound good in a sales offer.

For each of the five points I describe what was on the site, what the research says and how to check the same thing on your own site. At the end there is an order of actions and ways to measure.

In short

  • Most of the problems are ordinary technical housekeeping and classic SEO. Google says the same in its official guide from May 2026.
  • Content that is not in the HTML returned by the server is invisible to most of the crawlers of ChatGPT, Claude and Perplexity. Google copes with it.
  • The price and the duration should be at the top of the page, in plain text.
  • Blocks on AI crawlers often work without the owner knowing. Every crawler has to be checked separately, because each does something different.
  • An llms.txt file and “schema for AI” have no measurable effect on visibility today.
  • AI answers are random. A reliable measurement needs at least 7 repetitions of every question.

What this text is based on

I look at a site like a developer: I check what the server actually sends to a crawler, not what is visible in the browser. I set the results of that review against public research. I do not describe the effects of an implementation here, because there is no “before and after” measurement for this site.

I have three reservations about the research itself:

  • Most of it comes from companies that sell tools for tracking visibility in AI.
  • Almost all of it concerns the English-language market.
  • The systems described change quickly. Some findings from the summer of 2026 may already be out of date.

Where I draw my own conclusion, I say so.

1. Content that is not in the code

What I found. The list of trainers was drawn by a script in the browser. In the code the server sends at the start, there was not a single name.

What the research says. In December 2024 Vercel and MERJ published an analysis of AI crawler traffic on their network. None of the main AI crawlers executed JavaScript: neither OpenAI’s three crawlers, nor ClaudeBot, nor PerplexityBot. Some of them fetched script files, but did not run them. The exception was Gemini, which uses Googlebot’s infrastructure (Vercel).

Little had changed by the autumn of 2026. In tests on logs from September, ChatGPT-User, ClaudeBot, Claude-User and PerplexityBot still did not fetch scripts. There are isolated reports that since the end of September OAI-SearchBot has started rendering pages on some sites, but OpenAI has not confirmed this (GEO Toolbox overview). I treat it as a signal to watch, not a basis for decisions.

One distinction is easy to miss. Google processes scripts and says so in its own guide, so AI Overviews and AI Mode may see that list. The problem concerns assistants outside Google.

How to check on your own site. Open the page, press Ctrl+U to see its source, and search for a name, a price or another important fragment. If you see it on the screen but it is not in the source, a script draws it.

What to do about it. Key content should be in the HTML the server returns. Server-side rendering or prerendering gives you that. It is a task for a developer, not for a copywriter.

2. No answers to the simplest questions

What I found. “How long does the training take?” The answer came only in the FAQ near the end of the page. “How much does it cost?” Only an example with a subsidy.

What the research says.

  • Kevin Indig analyzed 18,012 verified ChatGPT citations. 44.2% of them came from the first 30% of a page’s content (Search Engine Land).
  • The RESONEO team took apart 1,200 ChatGPT answers from July 2026. In quick answers, the fragment describing a page was about 200 characters long and stuck to the H1 heading. Pages ChatGPT opened ended up cited in 74% of cases, and those it only found in 7% (Search Engine Land).
  • Ahrefs studied 15 million queries that ChatGPT itself sends to a search engine. The word “price” appeared in English-language queries more and more often: in 6.0% in August and in 8.9% in September 2026. The authors advise keeping prices and parameters in plain text on your own site (Ahrefs).

An assistant assembles its answer from fragments of pages. If a competitor has stated the amount and the number of hours outright, it is easier to cite the competitor.

This is not about artificially cutting text into “chunks for AI”. Google counts that among the unnecessary things (Google’s guide).

How to check on your own site. Read the first screen of the offer page as if you knew nothing about the company. The price, the duration, the format and the place should be stated there.

What to do about it. Right under the heading of the training page, write who it is for, how long it takes, how much it costs (even just “from…”), in what format and where it takes place. In plain text, not in a graphic and not in content loaded later by a script.

3. A crawler sent nowhere

What I found. The server redirected one of the AI crawlers, ClaudeBot, to the address 127.0.0.1. The others got in normally.

Where such blocks come from. A redirect to 127.0.0.1 by crawler name is an old rule from “bad bot” lists pasted into the .htaccess file (an example from the WebHostingTalk forum). ClaudeBot ended up on such lists in the spring of 2024, when forum administrators complained about the number of its requests (a thread on the XenForo forum).

A block can also sit in a layer the site owner never looks into:

  • Hosting. Search Engine Land described a case in which some of ClaudeBot’s requests got a 429 code. The authors ruled out, one by one, their own safeguards and Cloudflare. The block was imposed by managed WordPress hosting (Search Engine Land).
  • CDN. Since July 2025 Cloudflare has blocked AI crawlers by default on new domains. Since September 15, 2026, new domains have training and agentic crawlers blocked by default on pages with ads, while search crawlers remain allowed (Cloudflare).

One provider, several crawlers. This distinction changes the assessment of the whole finding. Each of the big companies has separate crawlers for different tasks:

  • OpenAI: GPTBot collects data for training models. OAI-SearchBot builds ChatGPT’s search index. ChatGPT-User fetches a page when a user asks about something, and according to OpenAI the robots.txt rules may not apply to it (OpenAI documentation).
  • Anthropic: ClaudeBot collects training data. Claude-SearchBot builds the search index. Claude-User fetches a page in response to a user’s question (Anthropic documentation).
  • Perplexity: PerplexityBot builds the index, Perplexity-User fetches pages in response to a user’s question.
  • Google: Googlebot serves both the ordinary results and the AI features. Google-Extended is only a token in robots.txt that concerns the use of content by Gemini. It is not a separate crawler and does not affect presence in the search engine.

What this means for my finding. The blocked crawler was the training one. Anthropic describes such a block as a signal not to include future content from that site in training data. If Claude-User and Claude-SearchBot get through, the site can still appear in Claude’s current answers.

That does not mean the block is irrelevant. Ahrefs analyzed 66.6 million answers from the free, logged-out ChatGPT: 14.4% of them involved queries to a search engine. For American users Semrush gives 34.5% (MediaPost). So most answers are built from what the model learned earlier. Whoever blocks training crawlers does not add their own site to that knowledge. That is my conclusion, not a research result.

How to check on your own site. A quick test from the terminal will show whether the server treats specific crawler names differently:

for bot in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User; do
  printf "%-18s " "$bot"
  curl -s -o /dev/null -w "%{http_code} %{redirect_url}\n" \
    -A "Mozilla/5.0 (compatible; $bot/1.0)" https://yourdomain.com/
done

A 200 code means the server delivered the page. Codes 301 or 302 with the address 127.0.0.1, as well as 403 and 429, mean a block.

This test has a limitation: it checks only rules based on the crawler’s name. Firewalls that verify IP addresses may treat a real crawler differently than your computer. The final answer is in the server logs:

for bot in GPTBot OAI-SearchBot ChatGPT-User ClaudeBot Claude-SearchBot Claude-User PerplexityBot Perplexity-User; do
  echo "== $bot"
  grep -i "$bot" access.log | awk '{print $9}' | sort | uniq -c
done

For the standard Apache and nginx log format you will see how many responses with which code each crawler received. Also look into the robots.txt file: a Disallow entry for a given crawler works like a block, only quietly.

What to do about it. Search crawlers and those acting on a user’s question are worth letting in. OpenAI states plainly that a site excluded for OAI-SearchBot will not appear in ChatGPT’s search answers. Training crawlers are a business decision. After a change in robots.txt, OpenAI’s systems need about a day to take it into account.

4. Clutter

What I found. 476 city pages, largely with the same text. 16 addresses from the sitemap returning a 404 error. A training course from June 2024 in three copies.

What the research says. In its guide to AI features, Google writes that mass-producing pages for query variants, done mainly to influence rankings or AI answers, violates its policies on scaled content abuse. It adds that a large number of pages does not make a site better (Google’s guide).

The second number comes from AirOps. ChatGPT fetches far more pages than it later cites: 85% of the sources fetched do not make it into the answer (AirOps). “Being found” is only the first stage. In my opinion, copies and outdated versions of the same offer make that choice harder, although I have not found a study that measures it directly.

How to check on your own site. Run the sitemap through any SEO crawler and count the addresses with a code other than 200. Then compare two random city pages and count the sentences that differ between them apart from the city’s name.

What to do about it. Keep city pages only where the company has something local to say: dates, a room, a trainer. Merge the rest and redirect. Remove from the sitemap the addresses that return 404. One training course is one page, and past dates are an archive, not copies.

5. The biggest asset was off the site

What I found. Over 150 Google reviews with an average of 5.0.

What the research says. The most often repeated number comes from AirOps: 85% of brand mentions in AI answers relied on third-party sites, 13.2% on the brand’s own site, and about 2% had no source (AirOps).

Before anyone builds a strategy on that, three things about this study:

  • It covers over 500 queries from the vendor-search stage, for example about the best CRM for sales, from B2B SaaS industries.
  • GPT-5, Claude Sonnet 4.5 and Perplexity Sonar were studied. Google was not in the sample.
  • Almost 90% of the external mentions came from roundups, comparisons and reviews. AirOps offers a product for winning such mentions, which is worth remembering when reading the result.

For a local company the picture looks different. BrightLocal checked 200,085 local searches in ChatGPT, AI Mode and AI Overviews. Companies’ own sites accounted for 42% of all citations (BrightLocal). The Google Business Profile accounts for 28.5% of citations and is the basic source for Google’s AI (BrightLocal on directories).

The type of question matters too. According to Ahrefs, for questions about facts ChatGPT cited official sources in 63.4% of cases, and for comparison questions in 46.4% (Ahrefs).

My conclusion: the question “which training company is the best” is answered mainly by third-party sites and reviews. The question “how much does this training cost and how long does it take” is answered by your own site, provided a crawler can read it. You need both.

Three remarks on this point:

  • Ahrefs calculated, on 75 thousand brands, that brand mentions on the web correlate with visibility in AI Overviews more strongly than links: 0.664 versus 0.218. This is a correlation, not proof of a cause (Ahrefs).
  • Google warns that artificially produced mentions do not help, because they are subject to the same anti-spam systems as the rest of the search engine.
  • The sources assistants draw on shift quickly. Between March and September 2026, citations of Reddit in American ChatGPT answers fell by 88%, and of Wikipedia by 80% (Ahrefs). A strategy based on last year’s ranking of “most cited sites” ages fast.

What to do about it. Complete the Google Business Profile and keep collecting reviews. Take care to be present in the roundups and comparisons that already exist in the industry. And fix points 1–3, so that your own site can answer questions about facts.

llms.txt and structured data

The site had an llms.txt file, which is much talked about now.

Ahrefs checked 137,210 domains. 28% of them had the file, that is about 38 thousand. Of that group, 97% did not record a single fetch in May 2026, either by a crawler or by a human. Among the fetches that did happen, most were made by SEO audit tools (21.7%), and AI search crawlers accounted for 1.1% (Ahrefs). Google writes that such a file is not needed to appear in its AI features.

The file does no harm and is quick to make. But I see no data that would justify paying for it as a separate service.

It is similar with structured data. Pages cited by AI have it almost three times as often as uncited ones, which looks like proof. So Ahrefs checked 1,885 pages that added JSON-LD and compared them with a control group. The number of citations in AI Overviews, AI Mode and ChatGPT practically did not change (Ahrefs).

The test covered pages that were already cited, so it does not say whether schema helps pages AI does not know yet. Google states that structured data is not required in AI features, but it is still worth having for rich results in the ordinary search engine.

Three things that change the picture

There is no single “AI”. Every assistant uses a different back end. Google’s AI features rely on Google’s index. The free ChatGPT draws mainly on OpenAI’s own index, and the paid Thinking mode on Google’s results for about 75%; according to that analysis Bing does not supply the index (RESONEO). According to industry analyses, Claude uses the Brave search engine (Profound). Visibility in one system does not automatically carry over to the others, so each has to be checked separately.

In Poland, AI Overviews have been available since March 26, 2025, and AI Mode since October 8, 2025.

Visibility is not traffic. In an Ahrefs panel covering 115,254 sites, AI assistants accounted for 0.43% of visits in September 2026, and search engines for 26.82% (Ahrefs). At the same time, AI Overviews lower the click-through rate of the first result by about 58% (Ahrefs). You can be cited and get no click.

Answers are random. SparkToro and Gumshoe asked 600 volunteers to put the same 12 questions. In almost 3,000 attempts, the same list of brands came back less often than once in 100, and the same list in the same order roughly once in 1,000 (Search Engine Land). I write below about what follows from this for measurement.

What I would do, in this order

  1. Check the logs and unblock the AI search crawlers.
  2. Move the trainers, the prices and the duration into the HTML returned by the server.
  3. Add under the heading of every training course: who it is for, how long it takes, how much it costs, where.
  4. Merge the city pages, fix the sitemap, remove the copies.
  5. Complete the Google Business Profile and verify the site in Bing Webmaster Tools, if only for the citation report.
  6. Take care of real mentions: roundups, comparisons, industry portals.
  7. Let go of llms.txt and “schema for AI” as separate services.

The first four points are one-off work. None of them is “ranking in AI” in the sense of a new, separate service.

Google says so plainly, in fact: from its perspective, optimizing for generative AI is still SEO. That is Google’s position on Google, but the data from ChatGPT points in the same direction. Pages in first place in Google were cited there 3.5 times more often than pages from outside the top twenty (AirOps).

How to measure it when the answer is different every time

Two methods are the simplest: a fixed set of questions asked every month, and visits from chatgpt.com in the site’s statistics. Both work, but each has a gap.

A fixed set of questions. A single answer is noise. The meaningful measure is the percentage of answers in which the brand appears, not its position on the list (SparkToro). Researchers from the University of St. Gallen recommend at least 7 repetitions of every question in every engine, and judging the trend in windows of 2–4 weeks (arXiv). It is worth asking the questions in the ordinary interface, because answers from the API overlap poorly with what a user sees (RESONEO).

In practice, 10 questions × 7 repetitions × 3 assistants gives 210 answers for one round of measurement. That is a lot for manual work. Fewer repetitions, however, mean that a difference between months cannot be told apart from chance.

Visits from ChatGPT. Count the utm_source=chatgpt.com parameter, not the referrer alone, because most clicks from ChatGPT arrive without one. Between October 8 and 15, 2026, the tag disappeared from some links and the visits landed in direct traffic, so it is worth marking that week in reports (Search Engine Journal). Since May 13, 2026, Google Analytics 4 has had a separate “AI Assistant” channel, but it relies on the referrer (TechWyse). Clicks from AI Overviews and AI Mode go into ordinary organic traffic.

It is worth adding four more to these two:

  • Google Search Console, the performance report for generative AI. Launched on June 3, 2026, and according to industry reports available globally since the end of August. It shows impressions of links to the site in AI Overviews and AI Mode, broken down by pages, countries, dates and devices. Clicks from these features cannot be separated out in it (Google).
  • Bing Webmaster Tools, the AI Performance report. Since February 2026 it has shown how many times pages were cited in Copilot and Bing’s AI answers, and which queries led to that. It does not show clicks (Practical Ecommerce).
  • Server logs. They show the visits of crawlers that analytics does not register. ChatGPT-User is one of them, that is a page fetched while answering someone’s question (RESONEO).
  • A field about the source of contact in the form, with the option “ChatGPT or another AI assistant”. It lets you see the clients who got a recommendation and came without clicking a link.

Commercial tools automate the asking of questions. It is worth asking the vendor how many repetitions stand behind each number, and whether it asks through the interface or through the API. Google also reminds us that no external tool has insight into its systems.

One more thing about “before and after” measurement: the effect comes with a delay. ChatGPT keeps copies of pages in a cache and can serve a version from over 90 days ago (RESONEO). A conclusion drawn a week after a change will be premature.

Back to the question about cost

Most of this list is ordinary housekeeping on the site and old SEO. That is why there is no single amount in answer to the question “how much does ranking in AI cost”. First you have to check what a crawler gets today, because that decides whether the work needed is a developer’s, an editor’s, or none at all.

The order looks like this: access for crawlers, content in the HTML, answers at the top of the page and order in the addresses. Only then mentions off the site and measurement in repetitions.

If you want to go through this list on your own site, write to me.

Sources

Official documentation:

Research and analyses:

Rollout dates in Poland (sources in Polish):

Have a similar problem?

Describe it in your own words — you do not need to know the names of technologies or prepare a specification.

Describe the problem