Cookie consent handling for web scrapers

A consent interface can affect a crawl in two ways: its text can enter the extracted content, or it can prevent the intended page from loading. Blocking a request, hiding an overlay, removing a DOM node, and submitting a consent choice are separate actions. Check which problem occurred before choosing a remedy.

The ePrivacy Directive — often called the "cookie law" — requires websites to get explicit consent before setting non-essential cookies¹. GDPR layered additional requirements on top: consent must be freely given, specific, informed, and unambiguous². The result, since roughly 2018, is that nearly every site serving European visitors shows a cookie consent dialog on first visit.

Diagnose the captured page

Compare the browser view with the HTML supplied to the extractor. A banner's text may remain in the DOM even when CSS hides it. A content-replacing wall may leave only a teaser or consent page. Scroll restrictions can interfere with interaction, although overflow: hidden does not categorically prevent programmatic scrolling.

DOM-based extraction can include consent text when it resembles content. A browser render may be necessary for a site's interaction flow, but rendering alone does not establish that the article loaded.

Identify the consent interface

Known script URLs and selectors are useful diagnostic clues. Examples include OneTrust's otSDKStub.js and #onetrust-consent-sdk, Cookiebot's #CybotCookiebotDialog, Quantcast's iframe-based interfaces, and TrustArc's #truste-consent-track. Site-specific implementations and renamed selectors also exist.

Nouwens and colleagues' CHI 2020 study analyzed consent interfaces on 680 scraped sites. Its roughly 58% figure concerns the five CMPs' share of detected CMP adoption, not 58% of every site in the UK top 10,000.³ It is historical evidence, not a current detection-coverage estimate.

Block requests and filter elements

Ghostery's Playwright integration applies network and cosmetic filters from configured lists.⁴ EasyList Cookie and annoyance subscriptions include consent-related rules.⁵ Network blocking can stop a matching script; cosmetic filtering hides matching elements. Hidden text can still be present when page.content() captures HTML.

This example initializes the ads-and-tracking preset:

import { PlaywrightBlocker } from "@ghostery/adblocker-playwright";
import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();

// Load filter lists and enable blocking
const blocker = await PlaywrightBlocker.fromPrebuiltAdsAndTracking(fetch);
await blocker.enableBlockingInPage(page);

await page.goto("https://example.com/article");
const html = await page.content();
// Extract from clean HTML -- no consent banner present

That preset contains EasyList and EasyPrivacy. fromPrebuiltFull adds cookie and annoyance lists. The example's final comment describes an intended result, not a verified absence of consent markup: inspect the captured HTML before extracting it.

Blocking a consent script can also disrupt a site's own loading flow. Apify's published account describes its investigation of cookie-modal handling; treat a particular integration as a documented approach, not a coverage guarantee.⁶

Interact through known rules

DuckDuckGo's autoconsent library describes CMP detection and interaction rules.⁷ Typical phases detect the CMP, identify a visible popup, then perform opt-out or opt-in actions. A simplified rule looks like this:

{
  "name": "onetrust",
  "detectCMP": [{ "exists": "#onetrust-consent-sdk" }],
  "detectPopup": [{ "visible": "#onetrust-banner-sdk" }],
  "optOut": [
    { "waitForThenClick": "#onetrust-reject-all-handler" }
  ]
}

Real flows can involve iframes, multiple panels, delayed loading, or unavailable controls. The time spent depends on those steps and their timeout settings; there is no general per-page latency figure.

The following example combines filtering with checks and button interaction. It illustrates a custom selector fallback rather than an autoconsent integration:

import { PlaywrightBlocker } from "@ghostery/adblocker-playwright";
import { chromium } from "playwright";

const browser = await chromium.launch();
const page = await browser.newPage();

// Layer 1: network-level blocking
const blocker = await PlaywrightBlocker.fromPrebuiltFull(fetch);
await blocker.enableBlockingInPage(page);

await page.goto("https://example.com/article", {
  waitUntil: "domcontentloaded",
});

// Layer 2: check if any banner survived
const bannerVisible = await page.evaluate(() => {
  const selectors = [
    "#onetrust-banner-sdk",
    "#CybotCookiebotDialog",
    '[id*="truste-consent"]',
    ".qc-cmp2-container",
    '[class*="cookie-banner"]',
    '[class*="consent-banner"]',
  ];
  return selectors.some((s) => {
    const el = document.querySelector(s);
    return el && el.offsetHeight > 0;
  });
});

if (bannerVisible) {
  // Try common "reject all" / "accept all" buttons
  const rejectSelectors = [
    "#onetrust-reject-all-handler",
    "#CybotCookiebotDialogBodyButtonDecline",
    '[class*="reject"]',
    'button[title="Reject All"]',
  ];

  for (const selector of rejectSelectors) {
    const button = await page.$(selector);
    if (button) {
      await button.click();
      await page.waitForTimeout(500);
      break;
    }
  }
}

const html = await page.content();

Before applying a selector broadly, confirm that it refers to the intended consent control. A short list of selectors cannot establish coverage of an unfamiliar site's interface.

Crawlee's helper

Crawlee exposes closeCookieModals() on Playwright and Puppeteer crawling contexts.⁸

Under the hood, it's based on the "I Don't Care About Cookies" browser extension — a community project that Daniel Kladnik maintained from 2012 until Avast acquired it in September 2022⁹. The extension stopped receiving meaningful updates after the acquisition, and forks like "I Still Don't Care About Cookies" picked up some slack.

closeCookieModals() requires the idcac-playwright package to be installed separately — Crawlee doesn't bundle it due to licensing concerns.

The helper, Ghostery filtering, and autoconsent are distinct mechanisms. Check the installed implementation and your target page rather than assuming one library wraps another.

Recover a content-replacing wall

A banner can coexist with article content; a wall can replace it. For a replacement, confirm that the recovered response contains the intended page before extracting. Merely removing the visible dialog may leave no article.

The EDPB's position is that cookie walls violate GDPR because consent obtained under the threat of losing access isn't "freely given"¹⁰. But enforcement is uneven across EU member states, and plenty of sites — especially news publishers with paywall-adjacent models — still use them.

Site-specific recovery can involve the site's consent action, a stored cookie, and a subsequent navigation. The following sample shows placeholder cookie values:

await page.context().addCookies([
  {
    name: "OptanonAlertBoxClosed",
    value: new Date().toISOString(),
    domain: ".example.com",
    path: "/",
  },
  {
    name: "OptanonConsent",
    value: "isGpcEnabled=0&datestamp=...",
    domain: ".example.com",
    path: "/",
  },
]);
await page.goto("https://example.com/article");

These values are not a reusable consent configuration. Cookie schemas, scope, and the site's loading sequence determine their effect. Reinspect the resulting page after navigation.

Markdownee's consent path

The Markdownee Actor and other crawler interfaces share a closeCookieModals setting, enabled by default. Browser setup uses Ghostery with explicit EasyList, EasyPrivacy, annoyance, and cookie-list inputs. Before extraction, the crawler also removes recognized residual CMP containers.

For a content-replacing wall, Markdownee attempts targeted recovery through the site's consent manager and its fetching channels. It does not use DuckDuckGo autoconsent or Crawlee's closeCookieModals() helper for that recovery. A wall that cannot be resolved fails the request instead of being treated as the requested article.

HTTP-only fetching does not execute CMP JavaScript, but server-rendered consent markup and redirects remain possible. Cheerio does not generally become a browser crawler; the adaptive crawler can escalate its HTTP branch where that branch is enabled. Browser-capable paths also apply the configured waits around recovery so newly loaded content has an opportunity to appear.

Use the npm library guide for interface options. Inspect failures and captures to distinguish a consent interruption from a content-selection problem.

The legal angle (briefly)

Auto-accepting or blocking cookie consent dialogs for data extraction doesn't change your legal obligations. If you're scraping personal data from EU-targeted websites, GDPR applies to your processing regardless of whether you clicked "accept" on the cookie banner². The consent banner governs the site's use of cookies on your browser — it has nothing to do with your right to scrape the page content.

That said, respecting robots.txt, not overwhelming servers with requests, and being transparent about your scraping activities are still good practice — and arguably more relevant to legal compliance than cookie consent handling.

Updated: September 7, 2026