How do I check if GPTBot is blocked?

Look for a Disallow rule matching the GPTBot user agent in your robots.txt at the site root, and check that no global rule shadows it. This checker reads robots.txt for every major assistant user agent — GPTBot, OAI-SearchBot, ChatGPT-User, Google-Extended, PerplexityBot, ClaudeBot, CCBot, Applebot-Extended — and returns a copyable Allow snippet.

Does allowing Googlebot allow AI crawlers?

No. Each assistant crawler is a separate user agent. Allowing Googlebot has no effect on GPTBot or Google-Extended, which must be allowed explicitly.

Back to home

Free tool · No sign-up

Free AI crawler checker

Enter a domain to see which AI assistant crawlers your robots.txt allows and which it blocks. If GPTBot or Google-Extended is disallowed, no amount of content or schema work will get you cited — this is the first thing to check.

Why this is the first check

A blocked assistant user agent is a hard stop: the page is never fetched, so schema, FAQ passages, and llms.txt never get read. It is also the cheapest thing on the list to fix — one file, one deploy. Sites usually block by accident, inheriting a template robots.txt or a CDN default rather than deciding to.

Once access is clean, the next questions are answer-readiness and structured data. The AEO scorecard covers those, and AI visibility tracking explains how to measure whether the change worked.

Questions

Which AI crawlers matter?

GPTBot and OAI-SearchBot (OpenAI), ChatGPT-User (live browsing), Google-Extended (Gemini and AI Overviews training and grounding), PerplexityBot, ClaudeBot (Anthropic), CCBot (Common Crawl, which feeds many models), and Applebot-Extended. Each is a separate user agent in robots.txt, so allowing Googlebot says nothing about the others.

How do I unblock AI crawlers?

Add explicit Allow rules for each assistant user agent in robots.txt at your site root, and make sure no global Disallow rule shadows them. Copy the snippet this tool generates, paste it into robots.txt, then re-run the check to confirm the file serves correctly.

Should I allow every AI crawler?

If you want AI assistants to cite you, allow the retrieval crawlers (OAI-SearchBot, ChatGPT-User, PerplexityBot) at minimum. Training crawlers such as Google-Extended and CCBot are a business decision — blocking them protects content from model training but can reduce how well models know your brand.

Does allowing crawlers guarantee citations?

No. Access is necessary but not sufficient. Assistants also need extractable answer passages, entity-clear JSON-LD, an llms.txt index, and third-party corroboration. Access simply removes the hard blocker so the rest can register.

Want the fixes written for you?

The free site check audits three of your pages and hands back an FAQ block, a schema snippet, a page draft, and a 30-day plan. No card, no sign-up.

Run the free site check

AI visibility resources

The guides, comparisons, and free tools behind AgenticSEO.

Best AEO tools (2026)Buyer's guide comparing nine answer-engine optimization vendors on tracking, execution, publishing, and pricing transparency.AI Visibility Tools MatrixMaintained matrix of measurement-only versus execution-layer platforms across twelve capabilities, with sources and a last-updated date.Profound alternativeWhere Profound stops at measurement and what an execution layer adds: generated schema, FAQ drafts, and approved publishing.Semrush AI Visibility alternativeSemrush reports AI mentions; AgenticSEO fixes the pages behind them. Feature-by-feature comparison with pricing notes.AgenticSEO alternativesAn honest list of alternatives to AgenticSEO and TGP Agentic SEO, with the buyer profile each one actually fits.AI visibility platform for agenciesClient-scoped websites, white-label AI visibility reports, per-client audit schedules, and API access for agency teams.Best AI visibility tools for agenciesWhat agencies should require: multi-client scoping, white-label output, publishing rights, and defensible measurement.AI visibility trackingHow to measure Share of AI Voice on a weekly cadence, with the formula, competitor set, and per-engine rollup.Auto-publishing SEO updatesHow approved SEO and schema fixes reach WordPress, HubSpot, Webflow, and GitHub-hosted sites without manual copy-paste.Fix AI visibility automaticallyThe seven-step audit-to-publish loop, what the agent does unattended, and which steps still need a human decision.Agentic SEO architectureWhitepaper on the six components of an agentic SEO system, from crawl diagnostics to publish verification.FAQ schema generatorPaste your questions and answers, get valid FAQPage JSON-LD you can drop straight into a page head.