AI crawler access in robots.txt critical
We check whether the site blocks GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
ChatGPT, Claude and Perplexity read sites differently from Google. We will check yours on 16 technical points and show what stops AI from citing you.
The check takes 1–3 minutes: we crawl the page and everything it depends on.
We check whether the site blocks GPTBot, ClaudeBot, PerplexityBot and Google-Extended.
We look for text in the raw HTML: AI crawlers do not run scripts.
We request the page as a browser and as an AI crawler and compare the answers.
We check for /llms.txt — a short map of the site that language models read.
We check that the home page answers 200 and does not hide behind a chain of redirects.
We check that HTTPS works, the certificate is valid and http redirects to it.
We look for structured data that tells an AI the type and author of the content.
We check title, description, canonical, lang and Open Graph.
We check for a single H1, a sane heading hierarchy and the main/article/header tags.
We weigh the amount of text, the presence of an update date and a named author.
How much of the page is real text versus markup — thin pages are hard for AI to read.
Whether an email, phone or address is visible — AI answers "how to reach them" from what is on the page.
Links to official social accounts tie the site to an entity AI already knows.
We count the share of images without an alt attribute — without it neither AI nor a screen reader knows what is pictured.
We check that the sitemap exists, opens and is listed in robots.txt.
We measure the response time, the size of the HTML and whether compression is enabled.