AI Crawler Glossary Definition

An AI crawler is an automated system that discovers or retrieves web content for AI-related uses such as training-data collection, retrieval indexes, answer generation, or model-connected search.

AI crawlers are not all the same: some collect training data, some support retrieval, and some behave more like conventional search crawlers. Site owners often manage them through access policies, robots directives, and monitoring.