HTTP or a headless browser: inspect what the page needs

A content extractor can only select material it receives. Before choosing a browser crawler, check whether the desired text is already in the HTTP response. If JavaScript or interaction supplies missing content, arrange that work before extraction.

Fetching and extraction are separate responsibilities. HTTP clients and browser automation obtain documents; a content extractor then selects useful regions from those documents.

Compare the response with the rendered page

Search the response for distinctive text from the page:

curl -s "https://target-site.com/article" \
  | grep -c "some phrase from the article"

A match shows that sentence is present, not that the entire article is complete. Compare several sections, tables, or other required elements. A zero match can reflect client rendering, a redirect, access restrictions, an encoding difference, or changed text.

Framework names do not settle this question. A Next.js, Nuxt, SvelteKit, or Astro page can supply server-rendered content, but its application may also load important sections later. Inspect the actual route.

Use HTTP when it supplies the required document

HTTP-only fetching avoids running page scripts and a browser's rendering machinery. It can be sufficient for static pages and server-rendered content. It still performs network I/O, decoding, parsing, redirects, and any configured request/session work.

Crawlee provides CheerioCrawler for HTTP fetching and parsing.¹ Feed the resulting HTML to an extractor when you need main content rather than page-wide markup. Inspect failed and partial results before expanding the crawl.

Use a browser for missing behavior

Browser automation can execute scripts and interact with a page. It can help when:

• Content is populated after the initial response.
• A permitted authentication flow requires browser interaction.
• A “load more” control or scrolling requests additional content.
• A site's response depends on browser-managed state.

A browser does not guarantee access or completeness. Configure waits for the content you need, bound them, and inspect the resulting document. When the application exposes a suitable documented API, direct API access may be another option.

Measure the whole workload

Resource use depends on page size, scripts, assets, concurrency, waits, browser reuse, and network latency. Chromium and Puppeteer issue discussions document particular memory problems, not universal memory-per-tab limits or page-per-second ratios.²³

Measure a representative sample with the same success criteria. Compare elapsed time, peak memory, requests, and recovered content. Report whether a timing includes fetching, browser startup, rendering, and extraction; otherwise two numbers may measure different work.

Likewise, broad JavaScript-size surveys describe their sampled pages, not the script cost of your target site.⁴ Use those surveys for context and local measurements for capacity decisions.

Request identity affects either approach

HTTP requests remain observable through network, headers, session state, and other signals. Browser automation has its own observable behavior. Matching a TLS fingerprint does not make a request indistinguishable from an interactive browser visit.⁵

Libraries such as curl_cffi implement browser-oriented TLS behavior, but that is one part of their request behavior rather than an access assurance.⁶ The cited discussions of browser and TLS fingerprinting describe mechanisms; they do not establish that one fetching method is universally harder to detect.⁷⁸

Combine fetching paths deliberately

A combined pipeline can use a browser where rendering is required and HTTP where the response is sufficient. Crawlee offers both browser and HTTP crawlers.¹ Keep content validation consistent across those paths so a faster response is not accepted merely because it is nonempty.

Markdownee uses Crawlee and Playwright around the offline Trafilaturacore engine. Core receives HTML; it does not make HTTP requests. Python Trafilatura's paper describes that project's extraction approach, not Markdownee's network performance.⁹

Markdownee's adaptive default renders each page. Set a positive renderingTypeDetectionRatio to enable detection for its HTTP path, or choose Cheerio explicitly for HTTP-only work. Account for the content that the selected path can obtain.

Extraction benchmarks evaluate another boundary: which text is retained from supplied pages.¹⁰¹¹ They do not prove that a browser is unnecessary or that a site will permit access. After choosing fetching, select the output representation and inspect the complete pipeline's results.

Updated: September 7, 2026