AI Crawlers List 2026: OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot and How to Allow Them

AI crawlers fall into three groups: training bots (GPTBot, ClaudeBot, CCBot), AI search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot) and user-triggered fetchers (ChatGPT-User, Claude-User, Perplexity-User). To appear in AI search answers, allow the search bots in robots.txt; blocking the training bots is a separate business decision and does not remove you from AI search.

Last updated: October 2026. Every user-agent below was checked against the vendor’s official documentation. We review this list quarterly, because AI companies add new bots several times a year.

Key takeaways

• Search bots drive AI visibility. If OAI-SearchBot, Claude-SearchBot or PerplexityBot can’t access your site, ChatGPT, Claude and Perplexity can’t cite it in search answers.
• Training bots are a separate choice. You can disallow GPTBot or ClaudeBot and still allow the search crawlers of the same companies.
• User-triggered fetchers may ignore robots.txt. OpenAI, Perplexity, Meta and Amazon say so in their docs; use your CDN or web server if you need hard blocking.

What are AI crawlers?

An AI crawler is a bot that an AI company uses to fetch web pages, either to train AI models, to build an AI search index, or to read a page live when a user asks a question. Like any web crawler, it identifies itself with a user-agent string in the HTTP header, and most respect robots.txt rules.

The difference between the three types matters, because each one affects your site in a different way:

The full AI crawlers list for 2026

This is the full list of major AI crawlers with the exact user-agent token to use in robots.txt. “Respects robots.txt” reflects what each vendor states in its docs.

Sources (official documentation): OpenAI, Anthropic, Perplexity, Google, Apple, Meta, Amazon, DuckDuckGo, Mistral, Common Crawl.

Bytespider (ByteDance) appears in many server logs, but ByteDance doesn’t publish official crawler documentation, so treat it as unverified and control it at the server or CDN level.

Is ChatGPT a web crawler?

ChatGPT itself isn’t a crawler, but OpenAI runs three bots for it. OAI-SearchBot builds the index behind ChatGPT search, GPTBot collects training data, and ChatGPT-User fetches a page when a user asks ChatGPT about a specific URL. OpenAI lets you control each one separately, so you can disallow GPTBot and still allow OAI-SearchBot to appear in ChatGPT search.

Google-Extended and AI Overviews: what actually controls Google AI features?

Google-Extended does not control AI Overviews or AI Mode. Google says it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”; it only covers Gemini model training and grounding in Gemini apps. AI Overviews and AI Mode use regular Googlebot crawling, with no extra requirements. To limit what Google shows, use nosnippet, data-nosnippet, max-snippet or noindex, and check performance in Google Search Console.

How to allow AI search crawlers in robots.txt

To allow AI crawler access for search while opting out of training, add separate groups for each bot to the robots.txt file at the root of your site. Here is a common setup:

# AI search and user fetchers: allow

User-agent: OAI-SearchBot

User-agent: ChatGPT-User

User-agent: Claude-SearchBot

User-agent: Claude-User

User-agent: PerplexityBot

User-agent: Perplexity-User

Allow: /

# AI training bots: block (optional)

User-agent: GPTBot

User-agent: ClaudeBot

User-agent: CCBot

User-agent: Google-Extended

User-agent: Applebot-Extended

User-agent: meta-externalagent

Disallow: /

Sitemap: https://www.example.com/sitemap.xml

To allow everything, simply don’t disallow these user-agents. To block all AI training bots but nothing else, keep only the second group. Remember that robots.txt rules are public requests, not access control: a scraper that ignores them still gets through.

How to whitelist OAI-SearchBot and other AI bots in your CDN or firewall

If your CDN or web application firewall blocks bots by default, whitelist AI search crawlers by their published IP ranges, not by user-agent alone. A bot challenge or a 403 response blocks the crawler even when robots.txt allows it.

1. Check your CDN settings. Cloudflare and similar providers offer AI bot toggles that can block AI crawlers site-wide; review them first.
2. Use the official IP lists. OpenAI (openai.com/searchbot.json, gptbot.json, chatgpt-user.json), Anthropic (claude.com/crawling/bots.json), Perplexity, Google, Apple, Amazon, DuckDuckGo, Mistral and Common Crawl publish IP ranges. Meta points to its AS32934 network.
3. Create an allow rule that matches both the user-agent and the IP range, so fake bots using a spoofed user-agent string stay blocked.
4. Test it. Request a page with each user-agent and confirm a 200 response, not a 403 or a challenge page.

Anthropic notes that blocking its bots by IP may not guarantee an opt-out, because it also prevents them from reading your robots.txt. Use robots.txt for policy and IP rules for security.

How to check if AI crawlers can access your site

Check AI crawler access in three places: robots.txt, server responses and server logs.

• robots.txt: open yoursite.com/robots.txt and confirm no group disallows the search bots you want. WordPress sites often generate a virtual robots.txt from an SEO plugin, so check the live file, not the settings screen.
• Server responses: fetch a URL with each user-agent and confirm the status code and that the content is accessible in the initial HTML. Many AI crawlers don’t render JavaScript, so content loaded only client-side may be invisible to them.
• Server logs: filter your web server logs by AI bot user-agents to see which bots visit, which URLs they fetch and how often. Verify suspicious hits against the official IP ranges.

Which AI crawlers are most active on websites?

On most sites, the AI bot traffic in server logs comes from a handful of major AI crawlers: OpenAI’s crawlers, ClaudeBot, PerplexityBot, meta-externalagent, Amazonbot and Bytespider. The mix depends on your niche and size, so check your own logs rather than relying on averages.

Two trends matter for access policies. First, AI search crawler and user-fetch traffic grows as more people use AI agents and AI platforms to research, so AI traffic increasingly means real readers, not just scraping for AI training.

Second, hosting platforms such as Vercel and many CDNs offer server-side bot protection, often with AI crawler rules or toggles, which can block AI web crawlers before your robots.txt is ever read. Review those crawler rules whenever your host changes its defaults.

Common mistakes with AI crawler rules

• Blocking all AI bots with one rule and losing AI search visibility along with training access.
• Relying on Google-Extended to stay out of AI Overviews. It doesn’t do that.
• Old blocklists. A blocklist written before 2025 misses Claude-SearchBot, meta-webindexer, Amzn-SearchBot and Mistral’s bots.
• CDN rules that override robots.txt, returning 403 to bots you meant to allow.
• Never checking blocked crawlers. Each AI bot may hit a different security rule, so test every crawler user-agent specifically, not just one.
• Content that needs JavaScript to render, so crawlers that read only the initial HTML see an empty page.

How Contadu helps once AI crawlers can access your content

Letting AI crawlers in is step one; being cited is step two. Contadu helps with the content side:

• Plan topics AI engines answer: the Content Strategy module groups keywords into clusters and shows content gaps against competitors, so you know which questions your site should cover.
• Write content that’s easy to cite: the Content Writer gives real-time semantic recommendations, so each page covers the entities and subtopics AI search crawlers expect.
• Keep teams aligned: SEO specialists, writers and editors work in one workflow, with unlimited team members and WordPress integration.

See how it works with a 7-day free Contadu trial.

FAQ

What are the top 10 AI bots?

The most common AI bots in 2026 are GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI), ClaudeBot and Claude-SearchBot (Anthropic), PerplexityBot (Perplexity), Googlebot with the Google-Extended token (Google), Applebot (Apple), meta-externalagent (Meta) and CCBot (Common Crawl). Bytespider also appears often in logs but has no official documentation.

Who are the big 4 AI agents?

In crawler terms, the four companies whose bots most affect AI visibility are OpenAI (ChatGPT), Google (Gemini, AI Overviews, AI Mode), Anthropic (Claude) and Perplexity. Each runs separate bots for search, training and live fetching.

What are the top 10 web crawlers?

Beyond AI crawlers, the most active web crawlers are classic search bots such as Googlebot, bingbot, Applebot, DuckDuckBot and YandexBot, plus SEO tool and social media crawlers. AI search crawlers are now a growing share of bot traffic on many sites.

Do AI crawlers follow robots.txt rules?

Most documented AI crawlers do, including GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot and CCBot. User-triggered fetchers are the exception: OpenAI, Perplexity, Meta and Amazon say their user fetchers may not follow all robots.txt rules, because a person requested the page.

Should I block AI crawlers?

Block training crawlers only if you don’t want your content used to train AI models; that choice doesn’t affect AI search. Don’t block AI search crawlers if you want to be cited in ChatGPT, Claude or Perplexity answers.

How can you tell whether an AI crawler is genuine?

Check the request’s IP address against the vendor’s published IP ranges. A user-agent string is easy to fake, so a request claiming to be GPTBot from an unlisted IP is likely a scraper.

How do I block AI crawlers on WordPress?

Edit robots.txt through your SEO plugin or a robots.txt plugin, adding disallow rules for the training bots you want to block. For hard blocking, use your host’s or CDN’s bot settings, and check the live robots.txt file afterwards.

Do I need an llms.txt file for AI crawlers?

No. AI crawlers rely on robots.txt, a sitemap and crawlable HTML. llms.txt is an unofficial proposal that major AI search crawlers don’t require.