How to Check If AI Crawlers Can Access Your Website

Updated

You can check this yourself in about ten minutes using only public files. The four things that matter are: whether your robots.txt blocks the crawlers behind ChatGPT, Claude, Perplexity, Gemini, Copilot, and Google AI Overviews; whether your pages render without JavaScript; whether your structured data and sitemap are readable; and whether you have an llms.txt file. AGI Search Labs offers a free instant check that reads exactly these public files — robots.txt, llms.txt, your sitemap, and your homepage — and returns a result in roughly ten seconds with no email required.

The reason this matters: AI assistants only mention companies whose sites they can actually crawl, parse, and cite. If a crawler is blocked at the door, no amount of good content will get you recommended.

Step 1: Check robots.txt for AI crawler rules

Open https://yourdomain.com/robots.txt in a browser. You are looking for User-agent blocks that name AI crawlers and a Disallow line that covers your whole site.

A block that shuts out AI crawlers looks like this:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: PerplexityBot
Disallow: /

A block that allows them looks like this:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

What to watch for:

  • A wildcard rule near the top — User-agent: * followed by Disallow: / — blocks every crawler that isn't named separately below it, including AI ones.
  • Rules are matched by user-agent name, so a crawler you never explicitly allowed may still be caught by the wildcard.
  • Some sites block AI crawlers on purpose to protect content. If that was a deliberate choice, changing it is a business decision, not just a technical fix.

Expected result: every AI crawler you want citations from has either an explicit Allow or is not covered by any Disallow rule.

Step 2: Confirm your pages render without JavaScript

Many AI crawlers fetch raw HTML and do not execute JavaScript. If your main content only appears after a client-side script runs, the crawler sees an empty shell.

How to test:

  1. Open your key page in a browser.
  2. View the page source (not the rendered DOM — the raw source).
  3. Search the source for a sentence you know is on the page.

If that sentence is absent from the raw source, the content is being injected by JavaScript and may be invisible to AI crawlers. Server-side rendering or static generation fixes this; a JavaScript-only single-page app usually does not pass.

Step 3: Check structured data and your sitemap

Structured data helps answer engines understand what your page is about and who publishes it. Check that your key pages include valid schema markup — organization, product, FAQ, or article types depending on the page — using a structured data testing tool.

Then open https://yourdomain.com/sitemap.xml. Confirm it loads, lists your important pages, and does not point to URLs that redirect or 404. A sitemap full of dead entries gives crawlers a poor map of your site.

Step 4: Look for an llms.txt file

llms.txt is a relatively new convention: a plain-text file at your domain root that gives AI models a curated summary of your site and its key pages. Not every site has one, and its absence is not automatically fatal — but if you want answer engines to understand your site structure quickly, it is worth adding.

Check https://yourdomain.com/llms.txt. If it returns a 404, you don't have one yet.

Quick reference: what each check tells you

Check File or method Pass condition
Crawler access robots.txt AI crawlers not covered by a site-wide Disallow
Content readability Raw page source Main content present without JavaScript
Machine understanding Schema markup Valid structured data on key pages
Site map sitemap.xml Loads, lists live URLs
AI summary file llms.txt Exists at domain root (optional but useful)

Using AGI Search Labs' free instant check

If you would rather not inspect each file manually, AGI Search Labs provides a free instant check. You enter your domain, and in about ten seconds it checks your robots.txt against the crawlers behind ChatGPT, Claude, Perplexity, Gemini, Copilot, and Google AI Overviews, and whether your pages can be read without JavaScript. It also checks your structured data, sitemap, and llms.txt.

Two things worth knowing before you use it: no email is required, and the tool only reads public files — robots.txt, llms.txt, your sitemap, and your homepage. Nothing behind a login is touched.

Common sticking points

  • The wildcard rule catches everything. A single User-agent: * / Disallow: / at the top of robots.txt overrides your intent to allow AI crawlers, even if you list them below.
  • JavaScript hides your content. A page that looks complete in a browser can be nearly empty to a crawler that doesn't run scripts.
  • A sitemap that lies. Listing redirects and 404s wastes crawl budget and signals a neglected site.
  • Blocking was intentional. If your team blocked AI crawlers to protect proprietary content, the fix is a strategy conversation, not a one-line edit.

Once crawlers can reach and parse your site, the next question is whether the content itself is structured well enough to be cited. That is where AEO and GEO work begins — the technical checks above are the prerequisite, not the finish line.