Measurement lab methodology

This page describes how we test consent management platforms (CMPs) before we publish a single result. Publishing the method first, with no findings attached yet, is deliberate: it lets anyone check our rules against our results later, instead of trusting that the rules were written to fit whatever we found.

What this measures, in one line: specified browser behaviour on test sites we control and on sampled public homepages. It does not measure, and we never state, whether any product, configuration or website complies with GDPR, the ePrivacy rules or any other law. See section 10.

1. Two separate studies, never mixed

We run two studies that ask different questions and are never combined into one number, one table, or one chart:

• The lab study asks how a CMP behaves on our own identical test site, under configurations we control and disclose. This is the only place we make a product-to-product comparison.
• The wild study asks what we observe on a sample of real, independently-run websites that use each CMP. It reports aggregates only — never a named site, and never a product ranking — because how a real site owner configured their CMP, not the product itself, dominates what we'd observe there.

Charts, tables and prose keep the two visually and verbally distinct throughout the site. A lab finding is never quoted as if it were a wild-study finding, or vice versa.

2. Lab setup

Every CMP in the lab is installed on its own copy of an identical WordPress site: same WordPress core, PHP, theme, and five content pages, including one page with a YouTube embed and one page with a form. Each site starts from the same golden image; only the CMP under test differs between sites. We record the WordPress, plugin and theme versions, the CMP's own plugin version and plan, a hash of the CMP's loaded JavaScript, the CMP's exported settings, the browser version, and the tracker-list versions used, for every run.

Where the lab sites live: on a dedicated test domain, cookietelemetry-lab.com, with one subdomain per CMP (for example <name>.cookietelemetry-lab.com), served over HTTPS from a server in Germany. These are test sites, not real businesses: they carry a visible "test site" notice, are markednoindex for search engines, collect nothing about real visitors, and are separate from this website. Because subdomains do not isolate cookies set on the parent domain, the harness also checks that no cookie is set on the parent domain and clears the entire browser context between runs.

Tags under test: Google Analytics 4 (GA4), Google Ads, the Meta Pixel, and a YouTube embed. These are test properties we control, not real advertising or analytics accounts collecting data about real visitors.

Insertion methods are tested and reported as a matrix, never merged into one figure, because how a tag gets onto a page changes what a CMP can and can't do about it:

• Hardcoded directly in the site's theme header — the case most "auto-blocking" marketing claims are about.
• Google Tag Manager, with no consent logic built inside the GTM container itself.
• Plugin-inserted (for example, an official analytics or pixel plugin), where scheduled.

Configurations are also published and labelled separately, never blended:

1. Default install — activate the plugin and complete the vendor's own setup wizard using its defaults, with no changes.
2. Vendor-documented setup — follow the vendor's own published documentation for blocking GA4/Meta/GTM and for Consent Mode, and archive the exact version of that documentation we followed.
3. Vendor-supplied configuration — if a vendor supplies its own configuration or licence after seeing its result (see "Right of reply" below), we re-run it and publish it as an attributed third configuration.

We do not attempt our own "best-effort" tuning of a vendor's settings beyond what its own documentation describes — an untrained tuning pass invites the fair objection that we configured a product worse than a customer would.

Control sites: before every batch of runs, we run a known-good site with no tags installed at all — it must produce zero findings — and a known-bad site with tags installed but no CMP and no consent gate at all — it must produce an exact, pre-documented set of findings. Any drift from either expected result halts publication of that batch until the harness or lab environment issue is found and fixed.

3. Which vendors, and on which plan

Vendors are chosen by WordPress relevance — active installs and web prevalence — balanced so that vendors we formerly had an affiliate relationship with are not given an advantage in which vendors get tested or how. The selection rule is published here before any result, and the list below is in alphabetical order, which is the only order we use for vendor lists outside a lab comparison matrix.

First batch (free tiers). Every CMP in the first report is tested on its free tier, and every result carries the label "free tier as tested" together with the plan name and plugin version. We have not bought any paid tier for this batch. The first batch is:

• Complianz (free plugin)
• Cookiebot (free plan — note: on this plan blocking is configured manually; the vendor's automatic blocking is a paid feature and is therefore not what the first batch measures)
• CookieYes (free plan)
• iubenda (free plan)
• Klaro (open source)
• Real Cookie Banner (free plugin)

Where a free tier does not include the feature that the vendor-documented setup (configuration 2) depends on, the result row says "not available on the free tier" rather than reporting a failure. A finding about a free tier is a finding about that free tier, on that date, and is never generalised to the vendor's paid product.

Paid tiers, only through disclosed reviewer licences.We ask vendors for a free, single-site reviewer licence for our lab site for the duration of the study. Every licence received is listed in section 12 below and beside the corresponding result. A granted licence adds that vendor's paid tier as its own labelled row; it does not replace the free-tier row and does not change how the product is tested or how the result is published. A refusal, or no answer, means the free tier only, and the report says so. The request itself carries no promise of coverage, timing or a favourable result.

Second batch. Vendors without a free tier, starting with Borlabs Cookie, and vendors whose terms are still being reviewed follow in a second batch under the same rules. A vendor that is not in the first batch is not being judged by its absence.

Pre-commitment: a former affiliate vendor's unfavourable result is published exactly like any other vendor's. The harness applies the same rules to every vendor; whether we have had a commercial relationship with a given vendor is attached only once we present the results, never earlier.

4. States we measure

Each lab run is one of the following states, always with a fresh browser context (no leftover cookies, storage or consent state from a previous run) and always from an EU location (an automated Chromium browser, Germany):

1. First load, no interaction.
2. Reject all, then measure and navigate to a second page.
3. Accept all, then measure and navigate to a second page.
4. A return visit after a reject, in the same reopened context — does the refusal persist?
5. First load with the browser's Global Privacy Control signal set, reported as a labelled synthetic diagnostic, not folded into the other states.

Timing is fixed and disclosed: we wait for the page's network activity to go idle (capped at 15 seconds) plus a further fixed 5 seconds before taking any action; after a click, we wait 15 seconds before moving on. Every recorded request is timestamped relative to the click that produced (or should have prevented) it. A request that lands within roughly a third of a second of a click is reported separately as "ambiguous" — close enough to the click that we cannot confidently attribute it to before or after — and is never counted as a finding either way.

5. Click verification and the three run outcomes

Clicking "reject" or "accept" in an automated browser can silently fail — the wrong element gets clicked, the banner reappears, or the CMP's own consent record never actually updates. So every run that involves a click is checked, and falls into exactly one of three outcomes:

• Valid run: the banner is confirmed gone, the CMP's own consent cookie or storage value decodes to the expected state using a per-vendor decoder we test independently, and — where Google Consent Mode is present — the page's dataLayer holds a matching consent update as supporting evidence (supporting only: a missing dataLayer entry on its own does not invalidate a run). Only a valid run can contribute an observation.
• Invalid run: the click happened, but verification failed — for example the banner reopened, or the consent state didn't decode as expected. This is reported as its own outcome (a case where reject was clicked but the recorded consent state did not update), never silently discarded and never merged into either a finding or an "automation failed" bucket.
• Automation failure: the run itself couldn't proceed — for example no matching banner selector was found at all. This is logged and reported separately from both of the above. It is never reported as a finding of any kind, including never as evidence of "activity after reject", since there was no verified reject to measure activity after.

Invalid-run and automation-failure counts are published alongside valid-run counts, so a low finding count can be checked against how often our own automation actually worked on that CMP's interface.

6. What counts as an observation

We use three observation classes, always kept separate:

• O1 — third-party tracking request before a choice: a request, made before any interaction, to a host on our tracker lists, that isn't allowlisted and isn't a Consent Mode denied-state (cookieless) ping.
• O2 — tracking storage before a choice: a cookie or localStorage key from a versioned identifier list, or set by a known tracker domain, written before any interaction.
• O3 — activity after a verified reject: an O1 or O2 event occurring after a click we verified (per section 5) as a successful reject, excluding anything that had already started before the click.

A finding counts only if the same result appears in at least 2 of 3 runs, on at least 2 different days, with the full evidence bundle stored (network log, screenshots before and after the click, list and script hashes) and one per-item human check. Anything seen only once, or only on one day, stays internal as "observed once, unconfirmed" and is never published as a finding.

What does not count as a finding, and is reported separately instead:

• A script-file download that carries no user data.
• A Google Consent Mode denied-state (cookieless) ping — reported as "a transmission with consent denied", not as tracking.
• A CMP's own consent-logging call to its own infrastructure ("CMP telemetry") — this is the CMP recording that a choice was made, not a tracker.
• A first-party request whose path or payload matches a known analytics or advertising pattern routed through the site's own domain (a "first-party collector candidate"). We report only that measurement traffic was routed via a first-party endpoint on that date — we do not, and cannot, observe what happens to it after that first hop.

"No traffic seen" is always written as "none observed in this window". It is never presented as proof that a product blocks a tag, or as proof of any particular mode of operation.

7. Tracker lists, licensed and attributed

Our primary tracker list is EasyPrivacy by the EasyList authors, licensed dual GPLv3-or-later / CC BY-SA 3.0-or-later; we use it under CC BY-SA 3.0 with attribution, applying its own rule syntax and exceptions rather than flattening it to a plain domain list. Our secondary list is our own curated tracker-endpoint list, covering the specific tags under test, with every entry sourced to that vendor's own documentation page. Both lists are version- or commit-pinned, and that pin is stored on every measured row. Our allowlist (the CMP's own domains and consent cookies, WordPress core cookies, the site's own static assets, self-hosted fonts) is published with the report, and each entry carries its own stated reason.

8. The wild study

The wild study observes real, independently-configured websites that use a given CMP, at the level of the registrable site's homepage only. Sites are discovered from public web-crawl datasets (used for discovery and sampling only, never republished) and confirmed by matching CMP loader signatures, then measured only in states 1 through 3 above, using the same classifier and the same confirmation rule as the lab (a counted finding needs a matching second run).

Visit budget. Each site receives at most 13 homepage requests per measurement cycle, and typically 4: one discovery cross-check, one request per measured state, and at most one retry per state per pass, plus a confirmation run for any site whose result feeds a published aggregate. We respect each site's robots.txt, never bypass a block, a CAPTCHA or an access barrier, never submit a form or click an advertisement, keep at most one connection to any host at a time, stop on objection, and send robots.txt checks and discovery fetches with a user agent that links to our bot information page, which carries an opt-out address. Measurement page loads use unmodified headless Chromium, whose user agent identifies it as automated ("HeadlessChrome") but not as ours; the bot page explains why and how to recognise these requests. (Corrected 5 October 2026: this sentence previously said all our requests carried the identifying user agent.) Respecting robots.txt is a courtesy we commit to; we do not treat it as anyone's permission.

From a wild-study visit we keep request hosts, paths and timing, cookie and storage names (not values), the detected CMP, the state and the date. We store no response bodies. Raw wild evidence is deleted within 60 days; only aggregates are kept, and a per-site mapping is never published or exported.

9. Statistics and presentation

• Every reported figure comes with its count, its eligible denominator, and confirmed/uncertain/invalid counts alongside it, plus a Wilson 95% confidence interval rather than a bare percentage, and the dates and conditions ("automated Chromium browser, Germany") under which it was collected.
• A per-CMP proportion in the wild study is shown only once we have at least 100 eligible sites for that CMP. Between 30 and 99, we show counts only, marked preliminary. Below 30, the figure is suppressed. Any downloadable breakdown (a "stratum") with fewer than 20 sites is suppressed the same way.
• Pooled, cross-CMP figures weight every CMP equally, regardless of how many sites we measured for it.
• We never rank CMPs against each other in the wild study. Wild-study CMPs are always listed in neutral, alphabetical order. Overlapping confidence intervals are never described as making two CMPs "equivalent", and an equal-weighted pooled figure is labelled a descriptive index — not a claim about market-wide prevalence.
• Our plain-language template for a wild-study figure: "Of 312 sites using a given CMP, between 41% and 53% loaded an ad or analytics tracker before the visitor chose. This reflects how sites set it up, not only the product."
• Only the lab study ever presents a comparison matrix, and even there we publish no overall score and no "best" or "worst". Findings describe the sampled homepages and the tested configurations, not EU-wide prevalence and not what a product causes.
• We explain how sites were selected, how many were blocked or excluded and why, and how often our CMP detection itself was wrong on a hand-checked sample.

10. What this measures, and what it doesn't

These tests measure specified browser behaviour, not GDPR or ePrivacy compliance. A match against our tracker lists identifies candidate tracking activity; a domain, host name or CNAME match alone does not establish what that request's purpose was. Cookieless transmissions and plain script downloads may still transmit information even though we don't count them as findings. Our exclusions and allowlist are measurement conventions we adopted to keep results comparable — they are not legal exemptions, and nothing on this site is a legal compliance determination for any product, configuration, or jurisdiction.

How we word a result. We write what was observed: "a request to host X was observed before any choice was made, in configuration Y, on date Z", or "did not block tag X in this configuration". We do not write that a product is illegal, violates the GDPR, is non-compliant, or "caused" anything, and we do not generalise a lab observation into a claim about the product in general or about real-world prevalence. Headlines, chart titles and any extract we share elsewhere follow the same rules.

Further limits. Our results are observations from an automated browser at specific dates and versions; vendors ship updates silently, and a script hash change triggers a re-run but a result can still be out of date by the time it is read. The lab tests one WordPress setup with a small tag set; it cannot speak for other platforms, other tags, or configurations we did not run. The wild study measures homepages only, and what a site owner configured dominates what we see there.

This content is for informational and educational purposes only and does not constitute legal advice. Consent management platforms, including those named on this site, are technology tools that assist with privacy compliance; each website owner remains responsible for complying with applicable law. Consult a qualified legal professional for advice on your situation.

11. Right of reply and corrections

Every vendor we test — affiliate or not, first batch or later — receives its own attributable lab and wild-study findings, this methodology, and its redacted evidence (captured network records, screenshots and configuration, with credentials and third-party content removed) at least 7 business days before we publish anything naming it, together with an offer to have us re-run the test against a configuration or licence it supplies. A material adverse change to a finding restarts that vendor's window. A reply of up to 200 words is printed next to that vendor's result, subject only to a privacy and legal check of the reply itself. We do not require a vendor's approval to publish, and silence is not treated as agreement.

Corrections. A claim we can no longer support is corrected or withdrawn promptly. Corrections are kept in a dated, public changelog on the affected page; the original text is retained (struck through, where practical) rather than quietly edited, and every vendor affected by a correction is notified of it. To report an error or ask for a correction, write to [email protected].

Evidence retention. Lab evidence behind a published finding is kept for at least 6 months after publication so that a disputed finding can be re-examined. Wild-study raw evidence follows the 60-day rule in section 8.

12. Conflict of interest and vendor assistance

Until 24 September 2026, CookieTelemetry was enrolled in the affiliate programmes of Termly, CookieYes, Complianz, Enzuzo and CookieHub. We earned $0 in commissions from any of them, and on 24 September 2026 we exited all five: we gave notice to end each enrolment and removed every affiliate link from this site. CookieYes's programme terms set a 30-day contractual notice period, which runs until 24 October 2026, with no affiliate links and no payments. Termination does not erase the historical conflict: these five vendors' results were designed and measured while we were enrolled, even though nothing was ever earned. See our independence statement for the full disclosure and dates.

This statement is shown before the first result on any page that presents lab or wild-study findings, together with which vendors ever paid us and which did not, any free product access we've been given, and any other vendor assistance. Each result row is labelled either "Former affiliate (enrolled until 24 Sep 2026; notice given; $0 earned)", with the contractual notice period added for CookieYes, or "No commercial relationship". Vendors did not, and do not, review or approve our findings before publication; each can reply, as described in section 11. Being tested, or how a vendor performs in a test, is never for sale — a commercial relationship (past or present) changes nothing about how a result is measured, verified, or published, and the harness is described as applying common rules to every vendor, not as "vendor-blind".

Free reviewer licences, if any, are listed here. We ask vendors for free single-site reviewer licences so that paid tiers can be tested without a purchase (section 3). Any licence, credit or other assistance received is recorded below with the date, and beside the result it applies to. As of 2026-09-24: none received. The free tiers used in the first batch are the vendors' publicly available free plans or plugins, obtained the same way any site owner obtains them.

Hinweis: Diese Website enthält keine Affiliate-Links mehr. Bis zum 24. September 2026 nahmen wir an den Partnerprogrammen von Termly, CookieYes, Complianz, Enzuzo und CookieHub teil, erhielten jedoch keine Provisionen, und haben an diesem Datum die Kündigung aller fünf Programme eingereicht.

13. Changes to this page

• 2026-09-24 — Version 1.0 published, before any result.
• 2026-10-05 — Correction to the user-agent sentence in section 8 (visit budget). Only robots.txt checks and discovery fetches carried our identifying user agent; measurement page loads on 23–25 and 28–29 September 2026 used unmodified headless Chromium's default user agent. Earlier wording: "… stop on objection, and identify our crawler with a user agent that links to our bot information page, which carries an opt-out address." See the bot information page.