Getting found
Crawler
A program that search engines and AI companies run to read websites page by page, following links as it goes. Also called a bot or spider. A robots.txt file on your site tells crawlers what they may and may not read.
Why it matters for your business
If a crawler cannot reach a page, that page does not exist as far as Google is concerned. Broken links, blocked folders and slow pages all limit what gets read. Letting the right crawlers in, and keeping the bad ones out, is basic upkeep.
In practice
If a hotel's booking pages get blocked in robots.txt during a redesign, bookings from search quietly drop until someone notices. One line in that file fixes it.
Related terms
Want this handled rather than explained?
Two minute intake. A real person reads every one and replies within a business day.