Get clean markdown for an LLM

Use this when you are feeding a page to a model: a RAG index, an agent's context window, a summariser. Raw HTML spends most of its tokens on markup, navigation and scripts. response_format=markdown returns the page's main article as markdown, with navigation, footers and asides stripped by default, and it parses PDFs to text as well. Markdown conversion costs nothing extra: you pay the engine price only.

Minimal request

curl https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/blog/pricing-update", "response_format": "markdown"}'

What comes back

The body is the markdown itself (Content-Type: text/markdown). Metadata is in the headers:

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
X-Engine: fetch
X-Target-Status: 200
X-Credits-Charged: 1
Cache-State: miss

# Pricing update for 2026

From 1 March, the Starter plan includes 50,000 requests per month.

| Plan | Price |
|---|---|
| Starter | $29 |
| Scale | $199 |

The HTTP status is the platform's. A page that answered 403 still comes back as HTTP 200 at 0 credits, so read X-Target-Status before you trust the content.

If you also set links, extract, autoparse, ai_extract, network_capture or screenshot, the response becomes the JSON envelope instead, with the markdown under content and an X-Warning: FORMAT_COERCED header. Set response_format=json yourself when you want the envelope from the start.

Options that matter

A project can store defaults for response_format, main_content_only, links and parse_pdf. A value in the request always wins.

Scope the page to cut tokens

include_tags and exclude_tags run before markdown conversion, so anything you drop never reaches your model:

{
  "url": "https://example.com/docs/install",
  "response_format": "markdown",
  "include_tags": ["article", ".docs-content"],
  "exclude_tags": [".cookie-banner", "aside.related", "pre.changelog"]
}

CLI: spicrawl scrape https://example.com/docs/install --format markdown --include article --include .docs-content --exclude .cookie-banner.

Token-saving checklist:

• Keep the default main-content isolation unless you need navigation text.
• Use include_tags for the one container you need. It is the largest single saving on documentation and news pages.
• Use text instead of markdown when your model does not need headings, links or tables.
• Leave links off unless you are crawling. Links arrive in a separate array; they do not inflate content.
• Leave cache on (the default). A cache hit costs 0 credits and returns the same markdown.

PDFs

When the target returns a PDF and you asked for markdown or text, the PDF is parsed to plain text and returned in the body at the normal engine price. Headings and tables are not reconstructed: a PDF has no HTML structure, so markdown and text return the same plain text.

• response_format=html returns the raw PDF bytes instead.
• extract, autoparse and links on a PDF fail with ERR::EXTRACT::FAILED, because a PDF has no DOM.
• An encrypted or scanned (image-only) PDF yields no text and fails with ERR::EXTRACT::FAILED; the detail says which.

Failure modes

A markdown body that is nearly empty with X-Target-Status: 200 usually means the content is rendered by JavaScript. Retry with js_render=true.

Cost

Markdown, text, include_tags, exclude_tags, links and PDF parsing add 0 credits. You pay the engine and proxy tier: 1 credit on the default fetch engine, 3 with js_render. Failures, cache hits and non-billable target statuses (anything except 200, 404, 410 or your allowed_status_codes) cost 0. X-Credits-Charged is the amount billed. See Credits.

Related

• JavaScript rendering when the markdown comes back empty
• Structured data when you want fields, not prose
• AI extraction Coming soon to have a model read the markdown for you
• Caching