Trafilatura and its forks & ports

Trafilatura is an open-source Python library and command-line tool for gathering text from the web. It extracts main text, comments, and metadata from HTML, with tools for discovering URLs, downloading pages, and exporting the results.¹

An HTML page is bigger than its article. Navigation menus, recommendations, advertisements, and footers can accompany the text a reader came for. In a research corpus, repeating those elements across thousands of documents introduces material that belongs to the website's template rather than the individual text being studied.² Stripping tags isn't enough.

Trafilatura tries to separate that main content from the surrounding boilerplate, while retaining useful structure such as paragraphs, headings, lists, and quotations. Its name comes from the Italian word for wire drawing — a fitting image for refining raw material into something usable.¹

Origins and history

Adrien Barbaresi started the work during a PhD at the intersection of linguistics and natural language processing. It was initially developed to build research text databases at the Berlin-Brandenburg Academy of Sciences, including work for its DWDS and ZDL units.³

The project grew out of earlier research on extracting news and blog articles. Barbaresi's 2021 ACL-IJCNLP system-demonstration paper presented Trafilatura as a reusable library and command-line tool, described its extraction process, and evaluated it on a collection of web documents; a lexicographic research platform needed usable text, and getting a download was only the beginning.²

I find that origin helpful: the project treats extraction as a decision about which text belongs in a corpus.

The original Python project now combines extraction with discovery, downloading, crawling, and output conversion. Its licensing also changed over time: version 1.8.0 moved to Apache-2.0; earlier releases used GPLv3+.¹⁴

How extraction works

Trafilatura works with an HTML tree parsed by lxml, or accepts one that the caller has already parsed. Its main extractor applies rules to the document's structure rather than requiring a custom scraper for each website.⁵⁶

First, it removes or simplifies unwanted elements and searches for likely content containers. An <article> element, itemprop="articleBody", and familiar IDs or classes such as entry-content provide clues. These are candidates, not guarantees: the extractor also prunes unwanted sections and tests link density, which helps distinguish a block of prose from a cluster of navigation links.⁶

Element handlers then collect paragraphs, lists, quotations, and tables while preserving supported structure. If the first pass captures too little, internal recovery looks more widely for text it missed. Other recovery stages can use a broader baseline or retry with more permissive recall settings, depending on the selected mode.⁷⁸

Outside fast=True, Python Trafilatura can also compare its result with bundled readability-lxml and conditionally try jusText. Fast mode skips those external comparisons; it still has internal recovery. That distinction matters when describing the descendants below.⁹⁸

Metadata extraction runs separately, reading titles, authors, dates, and other page information. Finally, the result is serialized: plain text is the default, with Markdown, HTML, JSON, CSV, XML, and XML-TEI available. The selected format determines how much of the extracted structure reaches the caller.⁷⁵

Forks and ports

A port reimplements software in another language; a fork develops inherited code in a separate project. Two related projects bring Trafilatura's extraction approach into other environments.

go-trafilatura, created by Markus Mobius, is a Go port. It extracts main text, comments, and metadata from supplied HTML. Its separate command-line tool can also fetch pages and process URL lists, feeds, and sitemaps.¹⁰

Trafilatura Core is an open-source extraction engine based on Trafilatura, implemented in TypeScript and native Python. Its TypeScript engine directly ports Python Trafilatura v2.2.0's fast=True path; go-trafilatura supplied only a DOM translation aid. Core's maintained native Python implementation translates that TypeScript engine. Both accept supplied HTML and return cleaned HTML with metadata and diagnostics, without fetching or rendering pages.¹¹¹²

Python Trafilatura is the source of the Go port and Core's TypeScript extraction engine. The dashed Go-to-Core relationship is only a DOM translation aid; Core's native Python implementation translates its TypeScript engine.¹⁰¹¹

For the extraction algorithms and recovery boundaries of Mozilla Readability, Python Trafilatura, and Core, see Trafilatura Core vs Mozilla Readability: how extraction works.

Updated: October 7, 2026