What kinds of AI Crawlers are there?

There are three types by function: copying back for training, building the AI index, and real-time retrieval. Crawling for training is intended to let models accumulate information. Crawling for indexing is intended to bring content into sources for later use. Real-time retrieval responds to a customer's question in the moment. A crawl does not mean the content has been used for training or indexed. Look at the function first, then at which company, because the same AI Platform may do several jobs. For how to tell them apart, see Figure 1-2, “AI Traffic Classification Tree”; for every company and crawler, see Figure 1-3, “AI Platforms and Crawlers Family Tree”; for which companies appear in each of the five traffic categories and how many visits every crawler made, see Figure 1-4, “IDAEO Traffic Atlas.” AI Agents act for customers and are counted separately from AI Crawlers.