Skip to main content

Markdownee npm library

Import markdownee into a Node.js application to fetch content, run a crawl, or work with stored results. The package also supplies the npm CLI.

Install and provision a browser

npm install markdownee
npx playwright install chromium

Use Node.js 22.22.2+ on the 22 line, 24.15.0+ on the 24 line, or 26+. Chromium serves adaptive and Chromium crawling; install Firefox with npx playwright install firefox when selecting playwright-firefox. Cheerio fetches over HTTP without a browser installation.

Crawl one URL

crawlOne(url, options?) returns a structured format map and follows no links. Its default format is Markdown; unavailable formats are omitted and request failure throws an error.

import { CrawlerType, crawlOne, OutputLayout, SaveFormat } from "markdownee";

const { markdown } = await crawlOne("https://example.com");

const contents = await crawlOne("https://example.com", {
  formats: [SaveFormat.Markdown, SaveFormat.MinifiedHtml, SaveFormat.Original],
  crawlerType: CrawlerType.Cheerio,
  outputLayout: OutputLayout.Standard,
});

The single-page options use the same camelCase field names as the crawl API, with these boundaries:

  • formats accepts txt, markdown, html, minified-html, and original, also exported as SaveFormat aliases and SAVE_FORMATS. The compact HTML result key is minifiedHtml.
  • crawlerType accepts playwright-adaptive, playwright-firefox, playwright-chromium, or cheerio. Short browser aliases belong to the CLI.
  • proxyConfiguration: { proxyUrls: [...] } accepts HTTP, HTTPS, SOCKS4, and SOCKS5 proxy URLs.
  • imageHandling accepts exclude, alt-text, or resolved-url. Return-only calls reject save; use CLI crawl-one with file output for downloaded images.
  • Link, table, and comment handling accept include or exclude. The former keep and strip values are not aliases.
  • save, storageDir, and includeHtml are excluded; use formats: ['original'] for captured input. Crawl-frontier, concurrency, and persistent-session options do not belong to the single-page call.

The excluded crawl fields include maxCrawlDepth, maxRequestsPerCrawl, maxResultsPerCrawl, globs, exclude, selector, useSitemaps, keepUrlFragment, initialConcurrency, maxConcurrency, deduplication, storeSkippedUrls, and sessionPoolName.

Readable html and compact minified-html share outputLayout. The default minimal uses body text and HTML fragments; standard adds ordinary metadata and complete HTML; enhanced adds allowlisted extended metadata and crawl information. Layout does not change captured original HTML.

Collect a crawl's results

Create an extractor, then pass starting URLs to run:

import { createCrawler, Deduplication, OutputLayout, Save } from "markdownee";

const extractor = createCrawler({
  save: [Save.HtmlKvs, Save.MinifiedHtmlKvs, Save.MarkdownKvs],
  deduplication: Deduplication.Minimal,
  outputLayout: OutputLayout.Enhanced,
  maxResultsPerCrawl: 10, // bounds the in-memory result set (0 = unlimited)
});

const { dataset, statistics, failures } = await extractor.run([
  "https://example.com",
]);
console.log(statistics, failures.map((failure) => failure.url));

await dataset.forEach((record, i) => {
  console.log(i, record.url, "depth:", record.crawl?.depth);
});
const all = dataset.export(); // LibraryRecord[]

The returned ResultDataset contains successful LibraryRecord values. Use dataset.export() to obtain the array or forEach to visit the collected records. Inspect failures for exhausted requests and statistics for request counts; these counts are not extracted-record or skipped-URL counts. Partial page failures preserve successful results. Invalid options and run-level errors throw.

Options use the JSON contract's camelCase names. Alias-object members serialize to the same raw string values. Library-specific options include includeHtml (default false), storageDir for also writing records to disk, and logLevel (default warning).

Construction validates and copies options. Each run owns its request queue, deduplication state, and logger, so repeated or concurrent calls can process the same URL. Without storageDir, records stay in memory. Explicit image save still persists image bytes in the CLI's resolved default storage; read their keys with KeyValueStore from markdownee/storage. Runs sharing a storage directory share persisted data. Dataset JSON/CSV export methods require an explicit store.

Import extraction functions, types, and vocabulary from markdownee; CLI construction from markdownee/cli; storage tools from markdownee/storage; and runtime validators from markdownee/schema. MarkdowneeLibraryInput and MarkdowneeCrawlOneInput validate their respective options using shared defaults. Importing these entry points does not execute a CLI command.

The library exposes navigation/request timeouts through its existing options. It has no public abort signal or streaming crawl interface; forEach visits results after collection. Python's async cancellation and per-child timeout belong to its subprocess adapter.

Optional markdownDiscovery uses off, alternate, negotiate, or probe to look for an origin-published representation. An accepted representation supplies all formats and adds markdownSource to the record. Links are still taken from available page HTML; when the page response itself is Markdown, there is no HTML link frontier. Discovery failures fall back to HTML extraction.

Invoke the CLI programmatically

buildProgram() returns the CLI's Commander program. Configure storage before executing a command or opening its stores:

import { buildProgram } from "markdownee/cli";
import {
  configureStorage,
  Dataset,
  resolveStorageDir,
} from "markdownee/storage";

const storageDir = resolveStorageDir();
configureStorage(storageDir);

const program = buildProgram();
await program.parseAsync([
  "node",
  "markdownee",
  "crawl",
  "https://example.com/",
  "--save",
  "markdown-dataset",
]);

const ds = await Dataset.open("default");
const page = await ds.getData({ limit: 100 });
console.log(`Extracted ${page.count} item(s)`);

A *-dataset save token places content inside each record. A *-kvs token stores the content separately and references its key. configureStorage(dir) sets Crawlee's directory, while resolveStorageDir() applies CLI-compatible resolution.

Export records to files

runExportAction reads the dataset index and writes available formats for successful records. It uses inline content or the referenced KVS value. Names derive from the title, then the URL, then page; manifest.json indexes the records.

import { runExportAction } from "markdownee/storage";

const result = await runExportAction({
  outputDir: "./markdownee-output",
  storageDir: "./storage",
});

console.log(`Wrote ${result.filesWritten} file(s) to ${result.outputDir}`);

ExportOpts accepts outputDir and storageDir. The ExportResult contains outputDir, filesWritten, recordsTotal, and manifestPath.

Remove local storage

runPurgeAction removes datasets/, key_value_stores/, and request_queues/ beneath the resolved storage directory. Like the CLI's purge command, this deletion has no confirmation prompt and cannot be undone by the library.

import { runPurgeAction } from "markdownee/storage";

const { storageDir } = await runPurgeAction({ storageDir: "./storage" });
console.log(`Purged ${storageDir}`);

PurgeOpts.storageDir is optional and uses CLI-compatible resolution when omitted. The call returns a PurgeResult containing the resolved storageDir; it does not call process.exit.

Open existing Crawlee stores

The markdownee/storage entry re-exports Dataset, KeyValueStore, and Configuration. These access storage from earlier runs without extracting pages:

import {
  configureStorage,
  Dataset,
  KeyValueStore,
} from "markdownee/storage";

// Point Crawlee's storage at a directory before opening any store.
configureStorage("./storage");

const ds = await Dataset.open("my-dataset");
await ds.forEach((item) => console.log(item));

const kvs = await KeyValueStore.open("default");
const value = await kvs.getValue("my-key");

Related guides

Updated: September 8, 2026