Bot traffic

Bot traffic is any request to a link made by software rather than a person: search engine crawlers building an index, the preview fetchers that messengers and social networks run when a link is pasted into a chat, uptime monitors, and scanners sweeping the open web. Short links live in public, so some share of this traffic is unavoidable, and much of it is wanted — a preview card exists only because a bot fetched the page behind the link.

Telling one kind from another is the hard part. A user-agent header is a self-declaration, and anyone can send any string, so a name on its own proves nothing. A verified crawler is confirmed the other way round: its address is matched against the ranges its operator publishes, or the address is resolved back to a hostname on the operator's domain and that hostname resolved forward to the same address. Traffic rules can then treat a confirmed crawler differently from anything that merely claims to be one, and reporting can keep human and automated visits apart instead of adding them together.