Get clean markdown for an LLM

Use this when you are feeding a page to a model: a RAG index, an agent's context window, a summariser. Raw HTML spends most of its tokens on markup, navigation and scripts. response_format=markdown returns the page's main article as markdown, with navigation, footers and asides stripped by default, and it parses PDFs to text as well. Markdown conversion costs nothing extra: you pay the request's normal price only.

Minimal request

curl https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/blog/pricing-update", "response_format": "markdown"}'

What comes back

The body is the markdown itself (Content-Type: text/markdown). Metadata is in the headers:

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
X-Target-Status: 200
X-Credits-Charged: 1
Cache-State: miss

# Pricing update for 2026

From 1 March, the Starter plan includes 50,000 requests per month.

| Plan | Price |
|---|---|
| Starter | $29 |
| Scale | $199 |

The HTTP status is the platform's. A page that answered 403 still comes back as HTTP 200 at 0 credits, so read X-Target-Status before you trust the content.

If you also set links, extract, autoparse, ai_extract, network_capture or screenshot, the response becomes the JSON envelope instead, with the markdown under content and an X-Warning: FORMAT_COERCED header. Set response_format=json yourself when you want the envelope from the start. An explicit json request gets the same main-content markdown under content, plus sections: the markdown split at its headings into {level, heading, text}, so an agent can read one section without the rest. Raw HTML is still available with response_format=html.

Options that matter

A project can store defaults for response_format, main_content_only, links and parse_pdf. A value in the request always wins.

Scope the page to cut tokens

include_tags and exclude_tags run before markdown conversion, so anything you drop never reaches your model:

{
  "url": "https://example.com/docs/install",
  "response_format": "markdown",
  "include_tags": ["article", ".docs-content"],
  "exclude_tags": [".cookie-banner", "aside.related", "pre.changelog"]
}

CLI: spicrawl scrape https://example.com/docs/install --format markdown --include article --include .docs-content --exclude .cookie-banner.

main_content_only also removes page chrome that sits inside the main article element: skip links, edit and toolbar widgets, navboxes, category and print footers, and nav or role=navigation blocks. On a MediaWiki article such as Wikipedia, that means the skip links, section edit links, toolbars and navboxes are dropped from the markdown. This removal runs only when main_content_only is on; include_tags and exclude_tags behave as before.

Token-saving checklist:

• Keep the default main-content isolation unless you need navigation text.
• Use include_tags for the one container you need. It is the largest single saving on documentation and news pages.
• Use text instead of markdown when your model does not need headings, links or tables.
• Leave links off unless you are crawling. Links arrive in a separate array; they do not inflate content.
• Leave cache on (the default) for faster repeat reads. A cache hit returns the same markdown and is billed at the same price as the fetch that stored it.

PDFs

When the target returns a PDF and you asked for markdown or text, the PDF is parsed to plain text and returned in the body at the normal price. Headings and tables are not reconstructed: a PDF has no HTML structure, so markdown and text return the same plain text.

• response_format=html returns the raw PDF bytes instead.
• extract, autoparse and links on a PDF fail with ERR::EXTRACT::FAILED, because a PDF has no DOM.
• An encrypted or scanned (image-only) PDF yields no text and fails with ERR::EXTRACT::FAILED; the detail says which.

Failure modes

A markdown body that is nearly empty with X-Target-Status: 200 usually means the content is rendered by JavaScript. Retry with js_render=true.

Cost

Markdown, text, include_tags, exclude_tags, links and PDF parsing add 0 credits. You pay for how the page was fetched and the proxy class: 1 credit for the default plain fetch, 3 with js_render. Failures and non-billable target statuses (anything except 200, 404, 410 or your allowed_status_codes) cost 0. A cache hit is billed at the same price as the fetch that stored it. X-Credits-Charged is the amount billed. See Credits.

Related

• JavaScript rendering when the markdown comes back empty
• Structured data when you want fields, not prose
• AI extraction Coming soon to have a model read the markdown for you
• Caching