For AI search, check that important public pages can be accessed, understood and indexed where relevant. Review robots.txt, page directives, hosting restrictions, internal links and accurate structured data. Search access and training permissions are different decisions. An llms.txt file is not a universal requirement or a guarantee of visibility.
The short version
- Diagnose a confirmed access issue instead of broadly removing security controls.
- Review each crawler's documented purpose and permissions separately.
- Keep visible facts and structured data consistent.
My site loads fine in my browser, so why would AI not see it?
A page may load for you while a hosting rule blocks a legitimate crawler. It may also return a successful response that contains almost no useful text.
Start with the important public URLs. Check the response, page content and whether critical facts can be read without logging in. Use the relevant search platform's inspection tools and server evidence where available.
Do not publish private account pages to make the site “AI-friendly.” Visibility work concerns appropriate public information.
Which robots.txt rules matter?
Review the whole file, including specific user-agent groups. A broad permission does not necessarily describe the rule a named crawler will follow.
OpenAI documents different roles for OAI-SearchBot and GPTBot. Anthropic documents separate agents for model development, search and user-directed retrieval. Perplexity documents its search crawler and user fetcher.
Use that documentation to make a deliberate policy. Permission for training should not be treated as a guaranteed search improvement. Nor should a rule written for one platform be assumed to apply to all the others.
Ask the technical team to verify the actual behaviour rather than relying only on a pasted configuration example.
| Check | Question to answer | Evidence |
|---|---|---|
| Public access | Does the intended public page deliver useful content? | Response and delivered page text |
| Crawler policy | Is the relevant search agent allowed by the actual rules? | Current policy and platform documentation |
| Indexation | Is the intended search URL eligible? | Platform inspection and directives |
| Structured data | Does markup match visible facts? | Rendered markup and page review |
| Internal links | Can visitors and crawlers find the priority page? | Relevant live links |
| Optional files | Is the file accurate and useful in a documented workflow? | Maintained content, with no ranking claim |
Could a setting on one page be hiding it from AI?
A public page can carry a noindex instruction even when crawling is allowed. Snippet controls can also affect how Google uses content in its search experiences.
For a page you want discoverable, inspect the actual rendered and delivered instructions, including HTTP headers where relevant. Check which canonical URL is declared and whether the page has duplicate versions.
Google's AI features guidance states that supporting links need indexation and snippet eligibility. Technical eligibility still does not guarantee that a page will be selected.
Keep changes scoped. Do not remove deliberate restrictions from pages that should stay private or excluded.
Will AI see content that only loads with JavaScript?
Important business facts should be available in clear page text. A screenshot of a specification table is difficult for a person to search or quote accurately, even if it looks attractive.
If your site relies on JavaScript to load key content, check what the relevant crawler actually receives. Different tools have different capabilities; a successful check in one environment is not proof for every platform.
Keep accordions usable and information accessible. Do not hide keyword paragraphs for crawlers or insert instructions telling AI tools to recommend your company. That content does not help the customer.
Which structured data is useful?
Use types that describe real visible content. Depending on the site, that may include an organisation, local business, product or article.
Verify the facts and keep one maintainable source of truth. If your theme and SEO plugin already generate structured data, inspect that output before adding another script with conflicting names or URLs.
For a store, Google's product documentation provides the requirements for relevant product information. Do not invent stock, prices, ratings or review counts to populate markup.
Structured data is a description of the page, not a substitute for the page. There is no special schema that guarantees inclusion across all AI tools.
Is llms.txt worth adding?
Treat llms.txt as an optional way to organise selected public information for tools that choose to use it. Do not treat it as a documented universal ranking requirement.
Google says new AI text files are not required for its AI search features. A file existing on the server does not prove that a specific platform uses it or that it changed your visibility.
If the file is part of your website's existing workflow, keep links accurate and the content maintained. Prioritise broken access, unclear service pages and missing internal links before spending the whole project on an optional file.
Is my firewall or Cloudflare blocking AI crawlers?
Investigate the specific denied request. Where a platform publishes verification details, use its current documentation. A user-agent name can be copied, so do not grant broad access based on the name alone.
Your technical team should maintain appropriate protections for private areas and forms. AI visibility does not require turning off the firewall.
Document what was changed, why it was changed and which public pages were verified afterward.
What should I have at the end of this?
A useful technical review ends with a list of confirmed problems and tested fixes: a priority page incorrectly excluded, conflicting product markup, a broken internal link or a legitimate search request being denied.
It should also distinguish “no issue found” from “not tested.” That makes later reporting honest.
Use our AI visibility audit guide to connect technical findings with answer observations. AdGrow's AI visibility services and free audit can help decide whether technical work is the first priority or whether the larger gap is content.
If you want the access and indexing side checked properly rather than guessed at, that is the technical half of what we do. See how we work on AI visibility, or request the free audit.
Frequently asked questions
Can I block training crawlers and still allow search access?
Some platforms document independent settings for these purposes. Review each platform's current agents and make the policy separately, rather than applying a blanket rule.
Does llms.txt make my website appear in ChatGPT?
There is no universal guarantee or requirement. Keep optional files accurate, but prioritise accessible public pages, useful content and the platform's documented search access.
Should I add more schema if I am not mentioned by AI?
First inspect the markup already generated and the visible page. More scripts are not a diagnosis. Fix inaccurate or conflicting information and improve the actual answer the page provides.
Sources and further checks
- OpenAI: independent crawler settings
- Anthropic: search, training and user retrieval
- Perplexity: crawler and access guidance
- Google: AI feature technical requirements
- Google: product structured data
Platform guidance checked on 12 October 2026. Practical recommendations and hypothetical examples are AdGrow’s editorial guidance, not platform guarantees.