Skip to main content

Markdownee Apify Actor

Use the Apify Actor for hosted URL extraction, configurable link-following crawls, and scheduled runs. Its dataset indexes the outcomes; content can be placed inline or in the key-value store for later API access.

Start a run

Open the Actor page, supply starting URLs, choose settings, and select Start. The command-line equivalent is:

apify call glueo/markdownee \
  --input='{"startUrls": [{"url": "https://example.com"}]}'

The Apify API accepts the same input:

curl -X POST "https://api.apify.com/v2/acts/glueo~markdownee/runs" \
  -H "Authorization: Bearer YOUR_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls": [{"url": "https://example.com"}]}'

These commands start Actor runs. An API token belongs in your execution environment, not in shared examples.

Read the input reference

The following tables retain the field names, types, and defaults used by the input contract. A blank default means no fixed schema default; the field's description explains omission behavior.

Crawl settings

ParameterTypeDefaultDescription
startUrlsarrayrequiredURLs to extract content from
crawlerTypeenum"playwright-adaptive"playwright-adaptive, playwright-firefox, playwright-chromium, or cheerio. Adaptive renders every page in a browser until renderingTypeDetectionRatio is raised above 0, then switches per page; cheerio is HTTP-only and does not execute JavaScript
renderingTypeDetectionRationumber0(Adaptive only) Ratio (0–1) of pages probed for rendering-type detection. The default 0 probes nothing, so every page is rendered in a browser; raise it to let adaptive fall back to plain HTTP on pages the probe finds static
markdownDiscoveryenum"off"Use a Markdown representation the site publishes for a page instead of extracting from its HTML: off, alternate (only when the page advertises one), negotiate (also ask for text/markdown), or probe (also try a .md sibling). Affects every output format, not only Markdown
maxRequestsPerCrawlinteger0Maximum requests to handle (0 = unlimited)
maxCrawlDepthinteger0Maximum link depth from start URLs (0 = unlimited)
initialConcurrencyinteger0Initial parallel pages/clients (0 = Crawlee default)
maxConcurrencyinteger3Maximum concurrent pages/requests; page size and browser work also affect memory
maxRequestRetriesinteger3Retries for failed requests
maxResultsPerCrawlinteger0Maximum results saved to dataset (0 = unlimited)

Crawl filtering

ParameterTypeDefaultDescription
globsarray[]Glob patterns matching URLs to include; each item is an object {"glob": "..."}
excludearray[]Glob patterns matching URLs to exclude; each item is an object {"glob": "..."}
selectorstring""CSS selector for links to follow
keepUrlFragmentbooleanfalseTreat URLs with different fragments as different pages
useSitemapsbooleanfalseDiscover and enqueue URLs from sitemap.xml at each start URL domain root
deduplicationenum"standard"minimal (Crawlee URL dedup only), standard (+ canonical URL, default), or aggressive (+ content hash)
respectRobotsTxtFilebooleanfalseHonor robots.txt rules

Content extraction

ParameterTypeDefaultDescription
modeenum"balanced"precision (less noise), balanced (default), recall (more content), or keep (no boilerplate removal, clean HTML only)
imageHandlingenum"exclude"exclude (default, remove images), alt-text (replace each image with its textual stand-in — no URL, no bytes), resolved-url (keep each image with its resolved absolute URL; nothing downloaded), or the save mode (download image bytes, re-encode, and store them in the key-value store — distinct from the save format-destination tokens below)
maxImageEdgeinteger1568Long-edge pixel cap for images stored by the save image-handling mode; never upscales; 0 = uncapped. Ignored unless imageHandling is save
rasterizeSvgbooleantrueStore SVGs rasterized to PNG in the save image-handling mode; disable to store the sanitized SVG source. Ignored unless imageHandling is save
linkHandlingenum"include"include (default, inline links) or exclude (keep the anchor text, drop the URL)
tableHandlingenum"include"include (default) or exclude (drop each table subtree, including its cell text)
commentHandlingenum"include"User-comment sections (forum/blog comments, not <!-- --> markup): include (default) or exclude detected comment containers
languageCodestring""Filter by declared language (e.g. "en"); empty disables the filter
outputLayoutenum"minimal"minimal (body-only TXT/Markdown and fragment HTML), standard (metadata/front matter and complete HTML), or enhanced (standard plus allowlisted extended metadata and crawl provenance); applies to both generated HTML variants

Output destinations

ParameterTypeDefaultDescription
savearray["markdown-kvs"]What to save and where, as format-destination tokens. Format: txt, markdown, html, minified-html, original (raw page HTML before extraction); destination: dataset (inline in the record) or kvs (a blob in the key-value store). List a format twice (e.g. markdown-dataset markdown-kvs) to save it to both. Saving generated or original HTML to the dataset risks out-of-memory on large pages — prefer kvs
datasetNamestringNamed dataset for results (empty = default run dataset)
keyValueStoreNamestringNamed key-value store for content files (empty = default)
requestQueueNamestringNamed request queue for pending URLs (empty = default)
storeSkippedUrlsbooleanfalsePush a dataset record for each URL skipped during crawling (auditing)

Browser settings

ParameterTypeDefaultDescription
headlessbooleantrueRun browser in headless mode
waitUntilenum"load"Navigation event: load (default), domcontentloaded, networkidle, or commit
navigationTimeoutSecsinteger60Navigation timeout in seconds
waitForDynamicContentSecsinteger10Maximum seconds to wait for dynamic content after navigation — the crawler continues as soon as the network is idle or this timeout elapses, whichever comes first (0 = disabled); also the timeout for waitForSelector and softWaitForSelector
waitForSelectorstring""CSS selector to wait for before extracting (request fails on timeout)
softWaitForSelectorstring""CSS selector to wait for before extracting (continues on timeout)
blockMediabooleantrueBlock images, stylesheets, fonts, PDFs, and ZIPs (Chromium only; on by default to cut memory and bandwidth)
ignoreCorsAndCspbooleanfalseDisable CORS/CSP restrictions
closeCookieModalsbooleantrueAutomatic cookie-consent handling: Ghostery-based ad/tracker blocking, accepting page-replacing consent walls via the site's own consent manager, and removing residual consent containers before extraction
maxScrollHeightinteger5000Max scroll height in pixels (0 = disable)
userAgentstring""Custom User-Agent string
ignoreHttpsErrorsbooleanfalseSkip SSL certificate verification

Proxy

ParameterTypeDefaultDescription
proxyConfigurationobjectApify proxy settings (use the proxy editor in Console)
proxyRotationenum"recommended"Rotation: recommended, per-request, until-failure
sessionPoolNamestringPersistent shared session pool name. Sessions (IP + cookies) are saved under this key and reused across Actor runs
maxSessionRotationsinteger10Maximum number of session rotations per request on block detection. Independent of maxRequestRetries. Set to 0 to disable

Cookies & headers

ParameterTypeDefaultDescription
initialCookiesarrayInitial cookie objects; cookie scope determines which requests receive them. Apify marks this input as secret
customHttpHeadersobjectCustom HTTP headers added to all requests

Understand the output record

Successful pages use status: 'success'. Exhausted request retries produce failed records with errors, retryCount, and crawl.scrapedAt; crawl.loadedUrl is included when available. With storeSkippedUrls enabled, skips use status: 'skipped' and skipReason.

A *-kvs route stores content separately and references its key and, when available, public url. A *-dataset route includes content in the record. Content nodes carry an MD5 hash and UTF-8 bytes count.

The example below illustrates record structure; its title, timestamps, sizes, and hashes are sample values:

{
  "url": "https://example.com/article",
  "status": "success",
  "metadata": {
    "title": "Article Title",
    "author": "Author Name",
    "date": "2026-01-15",
    "description": "Article description",
    "siteName": "Example",
    "languageCode": "en"
  },
  "crawl": {
    "loadedUrl": "https://example.com/article",
    "scrapedAt": "2026-04-11T12:00:00.000Z",
    "httpStatusCode": 200,
    "depth": 1,
    "referrerUrl": "https://example.com/"
  },
  "original": {
    "hash": "f8e6bd335e04d03e1be6798c2c72349c",
    "bytes": 89898
  },
  "markdown": {
    "hash": "c485356090a92c6a45e8c1155c14d8ee",
    "bytes": 4523,
    "key": "markdown-141fbc787408697a5d22735982be532a.md",
    "url": "https://api.apify.com/v2/key-value-stores/<storeId>/records/<key>"
  }
}

Selected formats appear as txt, markdown, html, or minifiedHtml. The original node carries the crawler capture's hash and size; an original-* route is needed to store its content. This capture precedes content extraction but may already reflect rendering, serialization, or consent handling.

Identify Markdown-sourced content

Enabled markdownDiscovery can accept a representation published by the origin. These records include markdownSource with the source URL and a mechanism of response, alternate, negotiated, or sibling.

verbatim is true when Markdown uses the served body after source front-matter removal and the selected output layout. It is false for Markdown converted from cleaned HTML or when no Markdown format was saved. The record's original is derived from the served representation on this path. Without markdownSource, content came from HTML extraction.

Discovery steps depend on crawler capabilities. Up to three alternates, one negotiated refetch, and one sibling may be attempted, with robots.txt checks and per-origin budgets also affecting requests. A rejected representation falls back to HTML.

Combine filters and destinations

This example sets starting URLs, include/exclude patterns, crawl limits, output routes, and Apify Proxy configuration:

{
  "startUrls": [{"url": "https://example.com/blog"}],
  "globs": [{"glob": "https://example.com/blog/**"}],
  "exclude": [{"glob": "https://example.com/blog/archive/**"}],
  "maxRequestsPerCrawl": 100,
  "maxCrawlDepth": 2,
  "save": ["markdown-kvs", "minified-html-kvs"],
  "mode": "precision",
  "outputLayout": "enhanced",
  "deduplication": "aggressive",
  "proxyConfiguration": {
    "useApifyProxy": true
  }
}

URL patterns filter candidate links; they do not enable link following by themselves. Add a selector such as a[href] when the run should enqueue links from the starting page, or supply the complete URL list.

Related guides

Updated: September 8, 2026