Markdownee for Python

Call the native Python library synchronously or asynchronously. Both languages share the source repository; Python uses Crawlee Python and Python Trafilatura Core and exposes no product CLI.

Install

Use Python 3.12 or newer:

python -m pip install markdownee

Tested PyPI platforms are macOS and Linux on x64/ARM64, and Windows x64. Native Windows ARM64 support is planned but currently blocked by impit packaging; 32-bit Windows (x86) is unsupported.

HTTP crawling needs no browser download. For browser crawling, install the selected Playwright browser:

python -m playwright install chromium
python -m playwright install firefox

Dependencies install separately. pyvips[binary] supplies libvips where supported; other platforms need a compatible libvips installation.

Extract one page

Save as extract.py, then run python extract.py:

from markdownee import fetch

contents = fetch(
    "https://docs.python.org/3/tutorial/introduction.html",
    crawler_type="http",
    formats=["markdown"],
)
print(contents["markdown"])

The program prints extracted Markdown. Fetch follows no links. Format selectors are txt, markdown (default), html, minified-html, and original; the compact HTML result key is minified_html. Unavailable formats are omitted, and a failed page raises MarkdowneeError. Select chromium or firefox for JavaScript rendering.

For an asynchronous application, save this as async_extract.py and run it:

import asyncio
from markdownee import afetch


async def main():
    contents = await afetch(
        "https://docs.python.org/3/tutorial/introduction.html",
        crawler_type="http",
    )
    print(contents["markdown"])


asyncio.run(main())

Crawl into files

Save as crawl.py and run python crawl.py:

from markdownee import crawl

summary = crawl(
    [
        "https://docs.python.org/3/tutorial/introduction.html",
        "https://docs.python.org/3/tutorial/controlflow.html",
    ],
    formats=["markdown", "html"],
    output_dir="./out",
    max_requests_per_crawl=2,
)
print(summary.succeeded, summary.failed, summary.manifest_path)

Each call writes files and manifest.json into a fresh directory below output_dir (default ./out). CrawlSummary exposes total/succeeded/failed/skipped counts and output/manifest paths. Partial failures retain successful files. acrawl() provides the same operation asynchronously.

Supported options

Link following requires a nonempty selector, stays on the seed origin, and obeys globs / exclude. Request/depth limits default to None; depth zero means start URLs only. Concurrent requests can slightly exceed the request limit.

Browser controls include headless, block_media, ignore_https_errors, wait_until, wait_for_selector, soft_wait_for_selector, wait_for_dynamic_content, and max_scroll_height. Durations use seconds. HTTP supports the required selector as a presence check and rejects browser waits/scrolling. Unknown options fail validation.

Images and layouts

Image save belongs to crawl() / acrawl(); return-only calls reject it. Saved rasters become metadata-stripped, first-frame WebP; SVG becomes PNG or cleaned SVG according to rasterize_svg. max_image_edge bounds raster dimensions without upscaling (0 disables resizing); SVG rasterization has a 1568-pixel edge ceiling. Images below 64 pixels on both edges are dropped. Failures retain their resolved URLs and add warnings. max_image_bytes (default 10 MiB) and max_image_pixels bound image processing; the default raster edge cap is 2048 pixels.

Derived outputs reference saved assets; original HTML is not rewritten. Selecting original also retains downloaded source image bytes. minimal gives body text and HTML fragments, standard adds ordinary metadata/front matter and complete HTML, and enhanced adds extended metadata/crawl fields. Serialization can differ by language.

Language differences

Python has URL deduplication but no adaptive selection, canonical/content deduplication, consent automation, Markdown discovery, sitemap seeding, custom cookies, persistent TypeScript storage, Apify destinations, or export/purge operations. Proxy schemes are HTTP and HTTPS. It accepts library keywords, not CLI save tokens. The playground identifies unsupported settings in generated Python examples.

Errors and limits

Catch MarkdowneeError for validation, crawl, and timeout failures; ProxySchemeError identifies unsupported proxies and MissingBrowserError missing browser setup. Inspect summary.failed after a crawl. max_input_bytes is checked after HTML capture, so it is not a network download limit.

Async cancellation stops awaiting a crawl; already-running extraction/image CPU work may finish. Synchronous calls belong outside an active event loop. See npm library Help for the TypeScript API and Apify Help for hosted execution.

Updated: September 26, 2026