CookieTelemetryBot

CookieTelemetry makes a small number of automated requests to public homepages to measure how cookie consent banners behave. This page explains what those requests look like in your server logs, what we keep, and how to opt out.

How our requests identify themselves

Two kinds of request come from us, both from our server in Germany.

• robots.txt checks and discovery fetches (plain HTML, no scripts) send the user agent CookieTelemetryBot/1.0 (+https://cookietelemetry.com/bot).
• Measurement page loads use an automated, unmodified headless Chromium browser. Its user agent contains HeadlessChrome (for example … HeadlessChrome/151.0.7922.34 Safari/537.36), and it reports itself as automated (navigator.webdriver). It does not carry our name or a link to this page. The page's own scripts, including the cookie banner, then make their usual requests. We do not put the word "bot" into this browser's user agent, because some consent platforms switch to a special mode for bots, which would change the very thing we measure.

Correction, 5 October 2026. Until today this page said that our requests carried the CookieTelemetryBot user agent. That was true only for robots.txt checks and discovery fetches. Our measurement page loads on 23–25 and 28–29 September 2026 used the headless Chromium user agent described above. Our measurement code did not set a user agent for them, and we found the gap in our own audit. The rest of this page (homepage only, the visit budget, robots.txt, no bypass of blocks, what we store, opt-out) was accurate for those visits. Earlier wording is kept in the corrections log at the end of this page.

Who we are

CookieTelemetry runs an independent research project measuring how consent management platforms (CMPs) — the tools behind cookie banners — behave in practice. Our full measurement methodology is published on this site. This crawler supports the "wild study" half of that work: aggregate statistics gathered from public websites that use a given CMP, never a report about any individual site.

What the crawler does

• Homepage only. We visit a site's homepage — the registrable domain's root page — and nothing else. We do not crawl internal links, and we do not follow a site beyond that one page.
• A published visit budget. Each site receives a small, fixed number of homepage visits per measurement cycle, covering discovery confirmation, the measurement states we run, and any confirmation or retry visits — at most 13 homepage requests per site per measurement cycle, and typically 4. We do not repeatedly hit a site outside that budget.
• EU/German egress. The crawler runs from a server in Germany, in the European Union — the same region whose visitor experience we're measuring.
• Robots.txt respected. We check and honour each site's robots.txt before visiting, and we never bypass a block, a CAPTCHA, or an access barrier — a block is recorded as a block, not worked around.
• No form submissions, no ad clicks. The crawler never fills in or submits a form, and never clicks an advertisement or any monetized link. It records what a page's own scripts do when the page loads and, in some measurement states, when its own cookie banner is clicked.
• One connection at a time. We keep at most one concurrent connection to any single host, and a low number of concurrent sites overall, so the crawl doesn't add meaningful load to any site we visit.
• We stop on objection. If a site blocks us, asks us to stop, or otherwise objects, we stop visiting it and record that, rather than retrying around the block.

What we store, and for how long

From a wild-study visit we keep only what's needed to classify which trackers loaded and when, relative to a consent choice:

• Request hosts, paths and timing (not full response bodies).
• Cookie and storage names (not their values).
• Which CMP was detected, and the measurement state and date.

We do not store response bodies or cookie values from wild-study visits. Raw per-visit records are deleted within 60 days of collection; only aggregate statistics (counts and rates across many sites) are kept beyond that, and a mapping from an aggregate figure back to any individual site is never published or exported.

Aggregates never name sites

Our published wild-study results describe a sample of sites using a given CMP as a group — for example, "of 312 sites using a given CMP, between X% and Y% loaded a tracker before a visitor chose." We do not publish, and do not intend to publish, a result that identifies which individual site produced which observation.

Opting out

To ask that we exclude your site from the crawl, email [email protected] with the domain you'd like excluded. We honour an opt-out request within one measurement cycle and keep a record of it so the site stays excluded from future cycles too. An opt-out also covers the verified-owner Scanner described below — an excluded domain cannot be scanned by either path.

The same bot, a second use: the verified-owner Scanner

The same two kinds of request (a robots.txt check that sends CookieTelemetryBot/1.0, then page loads in the unmodified headless Chromium browser described above) are also used by a separate, on-demand tool at cookietelemetry.com/scan: a free scanner a site owner can run against their own homepage, after proving they control the domain (a DNS TXT record, an HTML meta tag, or a confirmation link sent to an address at that domain). Unlike the wild study above, this path never visits a site unless someone who claims to control it has asked for the visit and proven it.

• Same rules. Homepage only, robots.txt honoured, no bypass of a block or CAPTCHA, EU/German egress, no form submissions or ad clicks.
• Rate-limited. At most 2 scans of a given domain and 5 scans requested from a given IP address in any 24 hours, and 60 scans total across all requesters in any hour.
• Private by default. A scan's result is shown only to whoever verified control of that domain — never published, indexed, or shown to anyone else.
• Same opt-out. A domain on our do-not-scan list (see "Opting out" above) is refused by this path too, even if someone claims to control it.

See the Scanner's own Terms of Service and Privacy Policy for the full detail on what it stores (a structured, already-classified summary — never a raw HAR, cookie value, or page recording) and for how long (results 7 days, a requester's email 30 days).

Corrections log

• 5 October 2026. Opening paragraph replaced and the section "How our requests identify themselves" added (see the correction above). The Scanner section, which said the Scanner used the same declared identity, now describes the same two kinds of request. Earlier wording of the opening: If you've seen a request in your logs fromCookieTelemetryBot/1.0 (+https://cookietelemetry.com/bot), this page explains what it is.