Cookie consent handling for web scrapers

A consent interface can affect a crawl in two ways: its text can enter the extracted content, or it can prevent the intended page from loading. Blocking a request, hiding an overlay, removing a DOM node, and submitting a consent choice are separate actions. Check which problem occurred before choosing a remedy.

Article 5(3) of the ePrivacy Directive covers storing information on, or accessing information already stored in, a user's terminal equipment. It requires clear information and consent, except where the operation is needed to transmit a communication or provide a service the user explicitly requested.¹ When consent is the basis, GDPR requires it to be freely given, specific, informed, and unambiguous.²

Diagnose the captured page

Compare the browser view with the HTML supplied to the extractor. A banner's text may remain in the DOM even when CSS hides it. A content-replacing wall may leave only a teaser or consent page. Scroll restrictions can interfere with interaction, although overflow: hidden does not categorically prevent programmatic scrolling.

DOM-based extraction can include consent text when it resembles content. A browser render may be necessary for a site's interaction flow, but rendering alone does not establish that the article loaded.

Identify the consent interface

Known script URLs and selectors are useful diagnostic clues. Examples include OneTrust's otSDKStub.js and #onetrust-consent-sdk, Cookiebot's #CybotCookiebotDialog, Quantcast's iframe-based interfaces, and TrustArc's #truste-consent-track. Site-specific implementations and renamed selectors also exist.

Nouwens and colleagues' CHI 2020 study analyzed consent interfaces on 680 scraped sites. Its roughly 58% figure concerns the five CMPs' share of detected CMP adoption, not 58% of every site in the UK top 10,000.³ It is historical evidence, not a current detection-coverage estimate.

Block requests and filter elements

Ghostery's Playwright integration applies network and cosmetic filters from configured lists.⁴ EasyList Cookie and annoyance subscriptions include consent-related rules.⁵ Network blocking can stop a matching script; cosmetic filtering hides matching elements. Hidden text can still be present when page.content() captures HTML.

This example initializes the ads-and-tracking preset:

import { PlaywrightBlocker } from "@ghostery/adblocker-playwright";
import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();

// Load filter lists and enable blocking
const blocker = await PlaywrightBlocker.fromPrebuiltAdsAndTracking(fetch);
await blocker.enableBlockingInPage(page);

await page.goto("https://www.bbc.com/news");
const html = await page.content();
// Inspect captured HTML for remaining consent markup before extraction.

That preset contains EasyList and EasyPrivacy. fromPrebuiltFull adds cookie and annoyance lists. Inspect the captured HTML before extracting it; filtering does not prove that all consent markup is absent.

Blocking a consent script can also disrupt a site's own loading flow. Apify's published account describes its investigation of cookie-modal handling; treat a particular integration as a documented approach, not a coverage guarantee.⁶

Interact through known rules

DuckDuckGo's autoconsent library describes CMP detection and interaction rules.⁷ Typical phases detect the CMP, identify a visible popup, then perform opt-out or opt-in actions. A simplified rule looks like this:

{
  "name": "onetrust",
  "detectCMP": [{ "exists": "#onetrust-consent-sdk" }],
  "detectPopup": [{ "visible": "#onetrust-banner-sdk" }],
  "optOut": [
    { "waitForThenClick": "#onetrust-reject-all-handler" }
  ]
}

Real flows can involve iframes, multiple panels, delayed loading, or unavailable controls. The time spent depends on those steps and their timeout settings; there is no general per-page latency figure.

The following example combines filtering with checks and button interaction. It illustrates a custom selector fallback rather than an autoconsent integration:

import { PlaywrightBlocker } from "@ghostery/adblocker-playwright";
import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();

// Layer 1: network and cosmetic filtering
const blocker = await PlaywrightBlocker.fromPrebuiltFull(fetch);
await blocker.enableBlockingInPage(page);

await page.goto("https://www.bbc.com/news", {
  waitUntil: "domcontentloaded",
});

// Layer 2: check if any banner survived
const bannerVisible = await page.evaluate(() => {
  const selectors = [
    "#onetrust-banner-sdk",
    "#CybotCookiebotDialog",
    '[id*="truste-consent"]',
    ".qc-cmp2-container",
    '[class*="cookie-banner"]',
    '[class*="consent-banner"]',
  ];
  return selectors.some((s) => {
    const el = document.querySelector(s);
    return el && el.offsetHeight > 0;
  });
});

if (bannerVisible) {
  // Try common "reject all" buttons
  const rejectSelectors = [
    "#onetrust-reject-all-handler",
    "#CybotCookiebotDialogBodyButtonDecline",
    '[class*="reject"]',
    'button[title="Reject All"]',
  ];

  for (const selector of rejectSelectors) {
    const button = await page.$(selector);
    if (button) {
      await button.click();
      await page.waitForTimeout(500);
      break;
    }
  }
}

const html = await page.content();

Before applying a selector broadly, confirm that it refers to the intended consent control. A short list of selectors cannot establish coverage of an unfamiliar site's interface.

Crawlee's helper

Crawlee exposes closeCookieModals() on Playwright and Puppeteer crawling contexts.⁸

Under the hood, it's based on the "I Don't Care About Cookies" browser extension — a community project that Daniel Kladnik maintained from 2012 until Avast acquired it in September 2022.⁹

closeCookieModals() requires the separate idcac-playwright package. Crawlee maintainers described that port in August 2026 as largely unmaintained and licensing-problematic while evaluating replacements.¹⁰

The helper, Ghostery filtering, and autoconsent are distinct mechanisms. Check the installed implementation and your target page rather than assuming one library wraps another.

Recover a content-replacing wall

A banner can coexist with article content; a wall can replace it. For a replacement, confirm that the recovered response contains the intended page before extracting. Merely removing the visible dialog may leave no article.

The EDPB's Guidelines 05/2020 conclude that an accept-or-no-access flow does not produce freely given consent, so consent obtained through that wall is not valid.¹¹

Site-specific recovery can involve the site's consent action, a stored cookie, and a subsequent navigation. The following sample shows placeholder cookie values:

await page.context().addCookies([
  {
    name: "OptanonAlertBoxClosed",
    value: new Date().toISOString(),
    domain: ".bbc.com",
    path: "/",
  },
  {
    name: "OptanonConsent",
    value: "isGpcEnabled=0&datestamp=...",
    domain: ".bbc.com",
    path: "/",
  },
]);
await page.goto("https://www.bbc.com/news");

These values are not a reusable consent configuration. Cookie schemas, scope, and the site's loading sequence determine their effect. Reinspect the resulting page after navigation.

Markdownee's consent path

The Markdownee Actor and TypeScript npm crawler/CLI share a closeCookieModals setting, enabled by default. Browser setup uses Ghostery with explicit EasyList, EasyPrivacy, annoyance, and cookie-list inputs. Before extraction, the crawler also removes recognized residual CMP containers.

For a content-replacing wall, Markdownee attempts targeted recovery through the site's consent manager and its fetching channels. It does not use DuckDuckGo autoconsent or Crawlee's closeCookieModals() helper for that recovery. A wall that cannot be resolved fails the request instead of being treated as the requested article.

HTTP-only fetching does not execute CMP JavaScript, but server-rendered consent markup and redirects remain possible. Cheerio does not generally become a browser crawler; the adaptive crawler can escalate its HTTP branch where that branch is enabled. Browser-capable paths also apply the configured waits around recovery so newly loaded content has an opportunity to appear.

The native Python library does not implement consent automation. Callers choose and apply any required consent interaction before relying on the captured page.

Use the npm library guide for interface options. Inspect failures and captures to distinguish a consent interruption from a content-selection problem.

The legal angle (briefly)

Handling a banner does not establish a legal basis for personal-data processing or grant content-reuse rights. GDPR territorial scope depends on the controller or processor and the processing described in Article 3, not merely on whether the scraped site targets EU visitors; covered processing also needs an Article 6 basis.¹²¹³ Assess access controls, terms, and applicable law separately.

robots.txt is a crawl-control protocol, not access authorization. RFC 9309 makes that boundary explicit.¹⁴

Updated: September 29, 2026