Extract fields from HTML with selectors, models, or both

Structured extraction turns source content into named values such as a product title, price, currency, and stock status. The work has two parts: locating the evidence and representing it in the expected shape. Valid JSON alone does not establish that a field is correct.

Choose a method according to the source structures, required fields, and cost of maintaining and checking the pipeline.

Address known structure with selectors

CSS selectors identify elements; XPath can also navigate relationships and use predicates. These examples assume that the expected nodes exist:

from bs4 import BeautifulSoup

soup = BeautifulSoup(html, "html.parser")

product = {
    "title": soup.select_one("h1.product-title").text.strip(),
    "price": soup.select_one("span.price").text.strip(),
    "in_stock": "Out of Stock" not in soup.select_one(".availability").text,
}

from lxml import etree

tree = etree.HTML(html)
price = tree.xpath(
    '//div[@class="product-info"]'
    '//span[contains(@class, "price")]/text()'
)[0]

A missing match needs handling in a complete application: select_one can return no element, and an XPath result list can be empty. Also check that a match is the intended value rather than an old price or an unrelated recommendation.

Selectors are inspectable and repeatable for a given document and parser. Site changes can invalidate them. Keep representative fixtures and validate required values so an empty or misplaced match does not become an accepted record.

Ask a model for a schema-shaped result

A model can locate fields from their meaning rather than from site-specific selectors. It still needs the relevant source content, a supported schema, and handling for missing or ambiguous evidence.

Firecrawl JSON mode

Current Firecrawl v2 places the schema inside a JSON format object: formats: [{"type": "json", "schema": {...}}]. Its older jsonOptions parameter is not part of v2.¹

This example uses firecrawl-py==4.41.0.² Install it with pip install firecrawl-py==4.41.0 and replace the placeholder API key and product URL before running:

from firecrawl import Firecrawl

app = Firecrawl(api_key="...")

result = app.scrape(
    "https://example.com/product/123",
    formats=[{
        "type": "json",
        "schema": {
            "type": "object",
            "properties": {
                "title": {"type": "string"},
                "price": {"type": "number"},
                "currency": {"type": "string"},
                "in_stock": {"type": "boolean"}
            },
            "required": ["title", "price"]
        }
    }],
)

product = result.json

JSON extraction's documented credit charge includes the base scrape and an additional JSON charge. One base credit plus four JSON credits totals five, not four times the basic request cost.³ Plan pricing and feature costs can change; use the current pricing page for budgeting.⁴

LangChain structured output

with_structured_output provides a schema-oriented calling interface, with behavior determined by the model, provider, and selected method. Pydantic schemas support automatic validation; TypedDict or JSON Schema results need their applicable validation path checked.⁵

from langchain_openai import ChatOpenAI
from pydantic import BaseModel, Field

class Product(BaseModel):
    title: str
    price: float
    currency: str = Field(default="USD")
    in_stock: bool

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
structured_llm = llm.with_structured_output(Product)

product = structured_llm.invoke(
    f"Extract product information from this page:\n\n{page_text}"
)

Distinguish provider-enforced schema output, tool calling, and response parsing. They have different failure handling. A supported schema constrains the representation; it does not prove that a price or stock status came from the right source evidence.⁶⁷

Combine extraction steps without losing fields

A main-content extractor can reduce page furniture before a model call, but it may also remove prices, SKUs, attributes, or other required fields. Inspect the intermediate result before adopting this arrangement.

Markdownee fetches pages and extracts with Trafilaturacore, then converts the selected content. The following example connects its CLI output to a model call:

import subprocess
from langchain_openai import ChatOpenAI
from pydantic import BaseModel

class Product(BaseModel):
    title: str
    price: float
    currency: str
    in_stock: bool
    description: str

# Phase 1: content extraction — the Markdownee CLI runs locally, no API call.
# `crawl-one` prints markdown to stdout; logs go to stderr.
clean_text = subprocess.run(
    ["npx", "markdownee", "crawl-one",
     "https://example.com/product/123", "--crawler-type", "cheerio"],
    capture_output=True,
    text=True,
    check=True,
).stdout

# Phase 2: LLM structuring on clean text
llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)
structured_llm = llm.with_structured_output(Product)
product = structured_llm.invoke(f"Extract product details:\n\n{clean_text}")

The example assumes the fetched content contains every required field. It also uses a placeholder product URL. For a working integration, supply a real source, inspect the intermediate content, and handle unavailable fields. No fixed token reduction, accuracy increase, or per-page price follows from adding this step.

Firecrawl's JSON mode likewise operates on a Markdown representation; its documentation says HTML attributes are unavailable to that extraction step.¹ Use source HTML when attribute values are required, or choose a method that explicitly carries them forward.

Define the result contract

JSON Schema expresses object structure, required properties, types, and constraints in a language-neutral form:

{
  "type": "object",
  "properties": {
    "title": {"type": "string", "description": "Product name"},
    "price": {"type": "number", "description": "Price in local currency"},
    "currency": {"type": "string", "enum": ["USD", "EUR", "GBP"]},
    "in_stock": {"type": "boolean"}
  },
  "required": ["title", "price"]
}

A Python Pydantic model can express validation and generate JSON Schema:

from pydantic import BaseModel, Field
from enum import Enum

class Currency(str, Enum):
    USD = "USD"
    EUR = "EUR"
    GBP = "GBP"

class Product(BaseModel):
    title: str = Field(description="Product name")
    price: float = Field(gt=0, description="Price in local currency")
    currency: Currency = Currency.USD
    in_stock: bool = True

These two examples deliberately have different constraints and defaults: the JSON example requires title and price, while the Pydantic example adds a positive-price check and default currency and stock status. They are not interchangeable contracts.

model_json_schema() exposes a model's schema for compatible consumers. Check which schema features a provider accepts, then validate returned values and missing-field behavior in your application.⁶

Compare the cost of a complete result

Measure fetching, parsing, model input/output, retries, validation, and maintenance separately. Model fees depend on token counts and the selected model's current rates.⁸ Service credits also depend on request features. A cost estimate is useful only when its input sizes, settings, and failure rates match the proposed workload.

For stable page structures, selectors may keep the pipeline small. For varied layouts, a model can reduce site-specific selector work but adds inference and verification. A combined pipeline is useful only if its early selection step preserves the evidence needed later. Crawl4AI also documents non-model extraction strategies.⁹

Updated: September 7, 2026