AI crawler and answer engine terms
Definitions of the answer-engine layer: answer engines, AI crawlers such as GPTBot, robots.txt permissions, knowledge graphs, and citation rate.
This is the vocabulary of the layer that classical SEO never covered: which agents fetch your pages, what permissions they need, how answers get grounded, and how citation performance is counted. These terms describe the mechanics behind every Share of AI Voice number AgenticSEO reports.
Answer engine
An answer engine is a search interface that returns a synthesized natural-language answer with citations, rather than a list of blue links.
ChatGPT Search, Perplexity, Google AI Overviews, You.com, and Microsoft Copilot are all answer engines. They retrieve candidate sources, synthesize an answer using a large language model, and cite (with varying fidelity) the sources that informed the answer.
Answer engines have changed the SEO problem: pages no longer need to merely rank — they need to be quotable. AEO and AgenticSEO are the disciplines that emerged to solve this new problem.
GPTBot
GPTBot is OpenAI's web crawler, used to gather data for training and grounding ChatGPT and related models. Sites that block GPTBot opt out of being represented in those systems.
OpenAI publishes three relevant user agents: GPTBot (training crawler), OAI-SearchBot (ChatGPT Search index), and ChatGPT-User (live in-conversation browsing). Blocking GPTBot in robots.txt prevents training-data ingestion but does not stop live browsing.
Most brands building AgenticSEO programs allow all three. The AgenticSEO platform writes a robots.txt template that explicitly allows GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended, and CCBot.
robots.txt
robots.txt is a plain-text file at the root of a website that tells crawlers which URLs they may or may not access.
robots.txt is the original web-crawler protocol, dating to 1994. It uses simple User-agent and Allow/Disallow directives. In the AI era it has become the front line of LLM access control: blocking or allowing GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and CCBot is done here.
A misconfigured robots.txt is one of the most common AgenticSEO failures. Disallowing AI crawlers makes the site effectively invisible to those assistants.
Knowledge graph
A knowledge graph is a structured map of entities — people, organizations, products, concepts — and the relationships between them, used by search engines and LLMs to ground answers.
Google's Knowledge Graph powers entity cards in search; LLM providers build their own internal knowledge graphs from training data and live web crawls. A brand with a well-formed Schema.org Organization entry, a Wikipedia page (where warranted), and consistent NAP (name, address, phone) signals across the web is more likely to appear as an entity in these graphs.
AgenticSEO contributes to knowledge-graph presence by emitting Organization + SoftwareApplication + Product JSON-LD on every relevant page.
Citation rate
Citation rate is the percentage of AI-assistant responses, across a defined prompt set, that mention or link to a given brand or page.
Citation rate is the core KPI of AgenticSEO. It is calculated as: (number of prompt responses that cite the brand) / (total prompt responses) across each AI assistant tracked.
A mature AgenticSEO program tracks citation rate by assistant (ChatGPT and Google Gemini), by prompt category (product, pricing, comparison, how-to), and over time. Increases in citation rate after a metadata, schema, or content change are the clearest signal that AgenticSEO work is paying off.
Related terms with their own page
More reference pages
See how these apply to your site. AgenticSEO audits every element on this page and publishes the fixes — run a free check, no sign-up.
Try the free preview