AI Platforms and Crawlers Family Tree: Which Companies and Which Crawlers Sit Under Each Function

Read from root to branch: function, AI Platforms, and crawlers; the same company on different branches means it takes part in different functions.

Crawler Family Tree
2026/3/3–10/8 · 123 million AI requests → purpose → AI Platforms → crawlers/automated browsing (60) · branch width = √requests
Root
Logged + separate browsing
Layer 1 · Purpose
7 categories, by function & audience
Layer 2 · AI Platforms
Dot = brand color/gray
Layer 3 · Crawlers/automated browsing (60 entries)
Capsule = name + requests; gray = function; sorted by requests
AI requests (all)
123 million
Unit: number of requests
Copies to train
S3
52.2M
42.3% of total
AI Crawlers
Builds search index
S4
41.26M
33.4% of total
Search crawlers + AI Overviews
Builds AI index
S4
3.99M
3.2% of total
AI Crawlers
Someone asks AI
S5
651,000
0.5% of total
Real-time retrieval (AI Crawlers)
Acts for customers
S8
18.09M
14.7% of total
AI Agents (incl. automated browsing)
AI Tools
7.08M
5.7% of total
Automated Bots · Info Services
Tracking
153,000
0.1% of total
Automated Bots · Info Services
Anthropic
Meta
OpenAI
Amazon
Tracking
Common Crawl
Microsoft
Google
Yandex
ByteDance
Sogou
Tracking
DuckDuckGo
Baidu
Brave
Yahoo Japan
Apple
Our AI Search
OpenAI
Perplexity
Parallel
Huawei
Linkup
Anthropic
You.com
Tracking
Kagi
OpenAI
Perplexity
Anthropic
DuckDuckGo
Google
Tracking
Mistral
Separately
TollBit
mcpbeat
Golemreach
Tracking
Andi
AgentTools-Cloud
Tracking
Google
DeepSeek
Tracking
Anthropic
Cohere
NICT
ROIS
ClaudeBot 24.48M
Copies back for training
meta-externalagent 11.58M
Copies back for training
GPTBot 10.6M
Copies back for training
Amazonbot 5.45M
Copies back for training
Reflectionbot 64,000
Copies back for training
Info service crawlers 307
Copies back for training
Other browsers 39
Copies back for training
CCBot 19,000
Copies back for training
bingbot 12.05M
Search index + AI Overviews
Googlebot 3.41M
Search index + AI Overviews
YandexBot 511,000
Search index + AI Overviews
Bytespider 256,000
Search index + AI Overviews
Sogou 253,000
Search index + AI Overviews
Other crawlers 31,000
Search index + AI Overviews
DuckDuckBot 12,000
Search index + AI Overviews
Baiduspider 4,630
Search index + AI Overviews
Bravebot 39
Search index + AI Overviews
Y!J-DLC 30
Search index + AI Overviews
Applebot 24.62M
Search index + AI Overviews
AI Search 2.81M
Builds AI index
AI Search External 5,918
Builds AI index
OAI-SearchBot 761,000
Builds AI index
PerplexityBot 218,000
Builds AI index
ShapBot 126,000
Builds AI index
PetalBot 114,000
Search index + AI Overviews
LinkupBot 60,000
Builds AI index
Claude-SearchBot 11,000
Builds AI index
YouBot 596
Search index + AI Overviews
Googlebot-Video 5
Search index + AI Overviews
Kagibot 4
Search index + AI Overviews
ChatGPT-User 635,000
Real-Time Retrieval
Perplexity-User 11,000
Real-Time Retrieval
Claude-User 2,494
Real-Time Retrieval
DuckAssistBot 1,624
Real-Time Retrieval
Google NotebookLM 464
Real-Time Retrieval
Gemini DeepResearch 10
Real-Time Retrieval
Python-urllib 19
Real-Time Retrieval
TABINOTE-ArrivalAudit 2
Real-Time Retrieval
curl 2
Real-Time Retrieval
HatchFeed 1
Real-Time Retrieval
MIN-SourceReview 1
Real-Time Retrieval
Meta-WebIndexer 1
Real-Time Retrieval
MistralAI-User 6
Real-Time Retrieval
AI Agents (browsing) 17.95M
Browsing (separate)
tollbot 126,000
Errands, address lookups
mcpbeat 7,925
Errands, address lookups
GolemreachTrustBot 5,683
Errands, address lookups
Real-browser AI Agents 5,102
Errands, address lookups
(separate)
Andi-AgentSurfaceProbe 1,956
Errands, address lookups
agent-tools.cloud-crawler 1,866
Errands, address lookups
AI tools 7.08 million
AI Tools
GoogleOther 142,000
Tracking in Progress
GoogleOther-Image 34
Tracking in Progress
DeepSeek 9,551
Tracking in Progress
HanaleiBot 574
Tracking in Progress
anthropic-ai 27
Tracking in Progress
Claude-Web 21
Tracking in Progress
CohereBot 30
Tracking in Progress
ICC Crawler 30
Tracking in Progress
Cotoyogi 2
Tracking in Progress
Logged 105,487,618 requests (2026/3/3–10/8) + AI Agents automated browsing 17,948,162 requests (2026/8/26–10/8) = tree root 123,435,780 requests.
Automated Bots = AI tools 7,079,310 requests + Tracking in Progress 152,613 requests; the Search Engine Crawlers' search indexing and AI Overview generation come from the same batch of requests.
Branch thickness follows the original figure, scaled to the square root of request counts (at least 1.5px); each row is attached only once. Capsule values are rounded in units of ten thousand / one hundred million.
ID
EO