Travel / news
Only 10 of 50 Major Travel Sites Give AI Crawlers Explicit Rules
A review of crawler policy files at 50 large travel websites found just one in five address AI bots directly, while the rest group them with ordinary crawlers or leave them on default settings. The gap is drawing attention as AI-referred visits to travel sites climb sharply.
The audit examined robots.txt files — the plain-text instructions that tell automated programs whether they may read a site and where they may go — across 50 major travel websites. Ten of them, or 20%, wrote rules that named AI crawlers specifically.
The other 40 either folded AI agents in with every other bot or left them to whatever the default settings allow, meaning the terms offered to AI systems are rarely a deliberate choice.
The review also found that stated directives do not always line up with which automated visitors actually get through, leaving policy and real-world access out of step.
The commercial stakes are rising. Adobe reported in July that AI-driven visits to travel sites jumped 119% year over year, as travelers increasingly use AI tools to shop and plan.
Vivek Pandya, an analyst at Adobe, described a "major bottleneck" in a blog post, writing that significant portions of U.S. travel and retail sites remain unreadable to machines, which restricts their presence across AI search results. When a large language model cannot easily read brand content such as amenities, pricing or availability, he added, potential revenue is being left on the table.
Travel brands have said publicly that they want AI systems to find them. Their own sites, according to the review, often signal something else — or nothing at all — a shaky starting point as the rules of AI engagement shift.