Trafilatura, readability-lxml, and Newspaper4k

These Python libraries overlap in article extraction but expose different workflows. Trafilatura combines content selection, metadata, downloading, and multiple output formats. readability-lxml returns selected HTML. Newspaper4k combines downloading, article parsing, and optional NLP.

Start with the result your application needs, then evaluate representative pages. A news article, product listing, and documentation page may expose different failure modes.

Installation and runtime

pip install trafilatura
pip install readability-lxml
pip install newspaper4k

Current package metadata lists Trafilatura 2.2.0 with Python 3.10+, readability-lxml 0.9 with Python 3.8.2 through versions below 3.15, and Newspaper4k 0.9.6 with Python 3.10+.¹²³ These versions differ from those in the historical benchmark below.

The stacks can include compiled dependencies such as lxml. They do not need a browser merely to parse supplied HTML. Newspaper4k's nlp extra supplies optional keyword and summarization support: pip install newspaper4k[nlp].

Markdownee uses Trafilaturacore, a separate pure-TypeScript engine. Python package requirements do not describe its extraction runtime.

Compare the calls

Trafilatura offers downloading separately from extraction and can serialize the selected content in several formats:

# Trafilatura
import trafilatura

downloaded = trafilatura.fetch_url("https://example.com/article")
text = trafilatura.extract(downloaded)
# or with options:
text = trafilatura.extract(downloaded, output_format="markdown",
                           include_links=True, favor_precision=True)

readability-lxml consumes HTML and returns an HTML summary with methods for title and other supported metadata:

# readability-lxml
from readability import Document
import requests

response = requests.get("https://example.com/article")
doc = Document(response.text)
title = doc.title()
html_content = doc.summary()  # returns HTML, not plain text

Newspaper4k uses an article object for download, parse, and optional NLP stages:

# Newspaper4k
from newspaper import Article

article = Article("https://example.com/article")
article.download()
article.parse()
print(article.text)       # plain text
print(article.authors)    # list of author names
print(article.publish_date)
article.nlp()             # optional — needs nltk
print(article.keywords)
print(article.summary)

For already-downloaded pages, Newspaper4k also supports set_html(). Its NLP call requires the relevant extra and resources; it is separate from article parsing.⁴

Outputs and integration

Python Trafilatura supports TXT, Markdown, HTML, XML, XML-TEI, JSON, and CSV.⁵ readability-lxml's summary() returns HTML, so a text-only consumer needs a conversion step; it also exposes title and supported author metadata.²

Newspaper4k exposes text, authors, publish_date, top_image, and movies, with keywords and summary available through optional NLP. Its CLI can export JSON and CSV.⁴⁶

The format guide explains what can be lost when HTML is converted to Markdown or text.

Read benchmark scores as versioned observations

The ScrapingHub article benchmark reports these versions and scores:⁷

Trafilatura has the highest F1 in these rows; Newspaper4k has higher precision and lower recall. This characterizes the evaluated article set, not the latest versions or all page types.

The Rust port separately reports ScrapingHub F1 0.966, precision 0.942, and recall 0.991. Its WCXB evaluation spans 2,008 pages across seven types, with F1 0.859 on the development split and 0.893 on the held-out test split.⁸ Keep the datasets separate. The port also adds classification behavior that is not part of Python Trafilatura or Trafilaturacore.

The SIGIR 2023 study compares extractors across eight datasets. It reports Python Trafilatura's macro mean F1 as 0.883 and JavaScript Readability's median as 0.970; the study's statistical comparison did not establish a significant mean-F1 difference between those two.⁹ JavaScript Readability and readability-lxml are distinct implementations. A project discussion about port synchronization is not proof that one always outperforms the other.¹⁰

Sandia's 2024 evaluation provides another dataset-specific comparison.¹¹ Neither that study nor another port's result measures Markdownee's current accuracy.

Test fetching and selection separately

A supplied-HTML extractor cannot recover content absent from its input. Capture JavaScript-rendered pages with a browser such as Playwright when needed, then pass that HTML to the selected library.

Consent interfaces can affect HTTP responses as well as browser views: a server can send banner markup, a teaser, or a redirect to a wall. Browser automation does not by itself solve authentication or paywalls. Markdownee adds Crawlee and Playwright fetching around Trafilaturacore, with bounded consent-recovery attempts; some requests still fail.

For extraction, inspect short pages, tables, code blocks, lists, and pages with several content regions. Trafilatura's favor_recall can retain more material, while fast skips fallback work. Those options trade work and selection policy; they do not imply a fixed speed multiplier.

Project lineage and licensing

Trafilatura was created by Adrien Barbaresi for text discovery and corpus construction.¹² Its license switched from GPLv3 to Apache 2.0 at version 1.8.0.

readability-lxml is maintained by Yuri Baburov and follows the Readability family of article-extraction tools.²

Newspaper4k is maintained by Andrei Paraschiv and is a fork of codelucas's newspaper3k.⁶ Keep package names and versions explicit when adapting older tutorials.

Choose with a small corpus

Include successful pages and known difficult structures. Compare required text coverage, unwanted content, metadata, and output shape before comparing elapsed time. Preserve the inputs and version/settings information so a future update can be assessed on the same examples.

Use Trafilatura when its broader output and collection tools fit, readability-lxml when selected HTML is the desired result, and Newspaper4k when its article object and optional NLP fit the application. None of those choices transfers a benchmark score to the separate Markdownee pipeline.

Updated: September 7, 2026