Bot Traffic and Click Quality: What Your Click Count Is Counting

Every redirect service can tell you how many requests it served. Almost none can tell you how many of those requests were people, which is the number every report is silently assumed to contain. Links live in the open web: they are indexed, previewed, scanned, monitored and scraped, and much of that happens within seconds of publishing, before a single human has seen the post.

This article is about the layer between the raw counter and a number you can defend in a client meeting: which automation you want to let through, why identifying a crawler by its name is not identification, what a network type proves, and how to turn that into rules and into an honest report.

The Raw Counter Is a Request Counter

A redirect engine counts what arrives. That is its virtue — it works for destinations you do not control, it survives blockers, and it counts a QR scan from a poster that has no session and no referrer, as covered in our guide to tracking link clicks. It is also why the number needs qualification.

The automated traffic that reaches a short link falls into recognisable groups:

• Link scanners. Email providers, messaging platforms and security gateways fetch links at delivery to check where they go — which is why a link accumulates clicks in the minute after a send, before delivery has finished.
• Preview fetchers. Social networks and chat apps fetch the link to build the card with the title and image. A well-behaved redirector answers these with metadata rather than routing and counting them.
• Search and SEO crawlers. Search indexers, plus commercial SEO tools that crawl anything they find.
• Monitors. Uptime checks, including your own. HEAD requests are served without counting a click for this reason.
• Scrapers and probes. Price collectors, harvesters, and scanners looking for links they can point elsewhere.

Two of those groups are traffic you asked for, one is neutral infrastructure, one is not welcome. "Block bots" treats them identically.

Good Bots, and the One Thing That Identifies Them

A user agent is a claim

The user agent is a header the sender writes. Anything can put any text in it, so a request calling itself Googlebot proves nothing, and a scraper calling itself Chrome is more common than one that admits what it is. User agent matching still earns its place — it catches automation that identifies itself honestly and corroborates other signals — but in the classifier it is explicitly a weak signal: it confirms, it does not decide.

Verification tests something the sender does not control

The address a request arrives from is not chosen by the sender in any useful sense, which makes it the right thing to test. There are two ways to do it.

Some operators publish the ranges their crawlers use. Google, Bing and Apple all do, and those files are pulled once a day into a fast lookup, so a request from a listed range is confirmed without network work at request time.

For crawlers without a published list — Yandex, DuckDuckGo, Baidu, Petal — verification is a two-step name check, and both steps are required:

Claim: "Googlebot" in the user agent

Step 1  reverse lookup of the address
        <address>  ->  crawl-<address>.googlebot.com   ends in the crawler's domain
Step 2  forward lookup of that name
        crawl-<address>.googlebot.com  ->  <address>    matches the original address
        verdict: verified crawler

Same claim, different address

Step 1  203.0.113.9  ->  host9.example-hosting.net     not the crawler's domain
        verdict: not verified, the claim is rejected

The forward step is not optional. Whoever controls an address block can usually set its reverse record to whatever they like, so a reverse name on its own can be fabricated. Resolving that name forward and getting the original address back is what closes the loop.

The CDN in front of the redirector also performs its own verified-bot categorisation, a verdict neither forgeable by the sender nor expensive to read. Confirmation is therefore taken in descending order of authority: published list, CDN verification, reverse-and-forward name check, then membership of a known spider range in the address database.

Why the name check never runs while a visitor waits

A reverse lookup is a network request with unpredictable latency, and the redirect is the one place where milliseconds are visible to a human, so it never runs synchronously. If a rule asks whether a crawler is verified and the answer is not cached, the rule treats it as unverified, the lookup runs in the background, and the answer is in cache for the next click from that address. Erring towards "not verified" costs one correct classification; erring towards a synchronous lookup costs a real person a delay on every request.

Network Type Is Context, Not a Verdict

Separately from what a visitor claims to be, the address range it came from has a known character. The commercial address database behind this classifies ranges into a small set of types:

| Type | What the range is | Share of IPv4 ranges in the database | | --- | --- | --- | | Not listed | Ordinary addresses, largely consumer and corporate | 72.1% | | Data centre, hosting, cloud | Servers, cloud instances, hosted browsers | 15.3% | | Public open proxy | Open relays anyone can route through | 9.2% | | Commercial VPN | Consumer and business VPN services | 3.2% | | Search engine spider | Ranges known to belong to crawlers | 0.17% | | TOR exit node | Exit points of the TOR network | 0.005% | | Web anonymiser | Browser-based proxy pages | 0.005% |

Read that table carefully, because it is easy to misread. Those are shares of address ranges in a reference database, not shares of anybody's traffic. One consumer ISP range serves enormous numbers of people; one hosting range may serve none. The table says what the categories are and how finely the address space divides among them, and nothing about what your campaign received.

The interpretive point matters more than the numbers. A data centre address does not mean a bot. Corporate networks route employees through cloud infrastructure, privacy-minded consumers use commercial VPNs for everything including shopping, and hosted browsers are a normal way to read the web. Delete every data centre click and you will delete real customers. Network type shifts confidence; on its own it is a fact about routing.

The Quality Score and What It Is For

The classifier produces two outputs: a visitor type — human, search crawler, library, data centre, VPN, TOR, or unknown — and a quality score from 1 to 100 with the reasons behind it. The inputs are the ones above plus behavioural signals: how often the same address hits the same link within a minute, whether the request carries a language preference at all, and whether the header set matches the platform the user agent claims.

The score is deliberately not a switch. Nothing is blocked because the score is low; the score explains a verdict and feeds the reports, while blocking is configured explicitly. The cost of a false positive is asymmetric: a bot counted as a human distorts a percentage, while a customer blocked as a bot never reaches the page and never tells you.

Type, score and reasons appear in the geography and traffic quality breakdowns, where the Networks table is the one media buyers reach for: autonomous system number, provider name, range type, total clicks and human clicks in separate columns. The gap between those last two, per provider, is usually the entire story of a traffic purchase. The full set is documented in geo analytics.

Turning a Verdict Into a Decision

A verdict is only useful if something can act on it. Traffic filtering rules attach an ordered list of conditions to a link, evaluated at the edge before the destination is chosen, with the first match deciding the outcome. A link with no rules performs no extra work at all.

The conditions relevant here are visitor type, verified-crawler status, network type and a quality-score threshold, alongside geography, autonomous system, address lists, device, language, referrer, user agent pattern, campaign tags, local time and first-versus-repeat visit. The outcomes: pass, block with 404, block with 410, redirect elsewhere, show a branded stub page, or pass while marking the visit suspicious.

Three design details decide whether this is safe to run in production.

Conditions are three-valued. Met, not met, or unanswerable because the data is missing — and unanswerable never counts as a match. If the address database is unavailable, a rule conditioned on network type does not fire and the visitor reaches the normal destination. Filtering fails open, never closed.

Inversion only flips confident answers. A rule written as "everyone except Germany" does not quietly fire on visitors whose country could not be determined. This is the most common way a well-intentioned geographic rule turns into an outage.

Pass exists so filters can be audited. A rule that lets traffic through and records that it matched is how you test a filter before letting it block anything: run it in pass mode, read the Filtered breakdown, confirm it catches what you expected, then change the action.

Put the general policy in the workspace-wide rule set, since a policy about data centres is rarely about one link, then use a link's own rules — evaluated first — to carry exceptions above the general ban. That is how your uptime monitor keeps working on the one link it watches. The caps on rules and address lists are in the traffic rules documentation.

One detail before you write a first-visit rule: telling a first visit from a repeat one requires a small cookie on your redirect domain, set only when a rule actually asks the question. That has privacy consequences, which is why it is opt-in by mechanism rather than by default.

What Changes in the Client Report

This is where the work pays for itself, and it is mostly about which numbers go on the page.

| Line in the report | Without a quality layer | With one | | --- | --- | --- | | Headline number | Clicks | Human clicks, with total clicks shown beside it | | Composition | Absent | Human, verified crawler, data centre, VPN, TOR, unknown | | Provider table | Clicks per network | Clicks and human clicks per network | | Filtered traffic | Invisible | Volume per rule, with the action each rule took | | Explanation of a spike | "Something happened on Tuesday" | The type and network the spike came from |

Two habits make that report honest rather than merely detailed. Show both numbers, always: a human-clicks figure with no total beside it invites the suspicion that the total was inconvenient. And report what you filtered and why — the rule identifier and action stored on every matched event mean the Filtered breakdown still reads correctly for rules you have since edited or deleted.

For agencies the provider table is also a commercial instrument: a network sending volume with almost no human clicks is a conversation with a supplier, and that conversation is easier with a per-provider breakdown than an aggregate. Our agency solutions and affiliate solutions pages cover how this fits into client reporting and payouts.

One thing to state plainly, because vendors in this category tend not to: classification is probabilistic. A determined scraper running a real browser from a residential address will be classified as human, and no product on the market will tell you otherwise. This layer removes the large, easy, honest-about-itself majority of automation and gives you evidence to investigate the rest. A report implying certainty will eventually be wrong in front of a client.

A Working Sequence

1. Leave bot filtering on by default in analytics, and treat any spike beginning before a human could have seen the link as automation until shown otherwise.
2. Read the composition breakdown before the headline number. An automated share unusual for the channel is itself the finding.
3. Check the Networks table for providers where total clicks and human clicks diverge sharply.
4. Write the first rule in pass mode, watch the Filtered breakdown for a week, and promote it to a blocking action only once the volume it matches looks like what you intended.
5. Keep verified crawlers passing, and exempt your own monitors with a link-level rule above the workspace policy.
6. Report human clicks and total clicks together, with filtered volume shown rather than subtracted silently.

The underlying idea is simple: a click is a request, a request has a provenance, and provenance is measurable. What makes it worth doing is the second-order effect. Conversion rates computed against human clicks are stable, geographic breakdowns without hosting ranges describe markets rather than infrastructure, and a client shown the composition of their traffic once stops asking whether the numbers are real.