Scrape a URL

Fetch one URL synchronously and return it as html, markdown, text, a PDF, or a JSON envelope with extracted data.

Requires scope: scrape

Credits (credits per successful request)

engine	direct	datacenter	residential	mobile
fetch	1	1	10	10
js_rendering	3	3	25	25
stealth_rendering	25	25	25	25
full_browser_rendering	8	8	32	32

ai_extract Coming soon adds 4. Failures cost 0; cache hits cost the price of the fetch that stored it. The billed amount is in the X-Credits-Charged response header, and the monthly allowance left after it in X-Credits-Remaining.

Fetch one URL synchronously and return it as html, markdown, text, a PDF, or a JSON envelope with extracted data.

How a request is rendered (cheapest first). You do not choose: Spicrawl picks the cheapest path that can do what the request asks.

Default (plain fetch, 1 credit): plain HTTP, no JavaScript, presenting a current browser TLS/HTTP2 fingerprint (impersonate, on by default and free; impersonate=false turns it off).
js_render=true (JavaScript rendering, 3 credits): content is built by JavaScript, or you need wait/wait_for/block_resources/network_capture.
Screenshots, response_format=pdf and actions use full-browser rendering (8 credits). stealth=true (full-browser rendering hardened against bot detection, 25 credits) is coming soon.
mode=auto: unsure - escalates through the available rendering tiers, bills only the tier that worked.
Managed proxy pools (premium_proxy) are coming soon; during the beta pass your own proxy with proxy.
The engine and headless fields are deprecated and ignored. Browser-only flags on the fetch tier are a 400, never silently ignored.

Billing: failures cost 0 (errors, bot challenges, timeouts, target statuses other than 200/404/410 unless listed in allowed_status_codes, undelivered screenshots). A cache hit is billed at the same price as the fetch that stored it: the cache saves the fetch (latency and load on the target), not credits. Like any request, a hit is refused with 402 when the monthly allowance cannot cover its price. Check X-Credits-Charged.

Bot challenges: a challenge page attributed to a named vendor (Cloudflare, DataDome, PerimeterX, Akamai) is never returned as a success. A fetch through a managed pool exit that meets one is retried once on a different exit, free (GET/HEAD/OPTIONS only, not with sticky_key or session_id, never on your own proxy). When you bring your own proxy (residential works best) and send no mode=auto, a request whose page is still a challenge is escalated automatically to stronger full-browser rendering on the same exit, which works through the interstitial and returns the real page; there is no flag to set. This is how Cloudflare-protected pages are supported. Automatic handling on managed premium exits is coming soon. Success is not guaranteed for every page. Only the tier that returned the page is billed (up to 25 credits, or the first tier's price), and 0 if every tier was challenged (502 ERR::UPSTREAM::CHALLENGE). On the default shared pool a challenged page may return that error; retry with your own residential proxy. A transport error, a refusal with no vendor marker (such as a bare 403) or a non-billable status such as 404 does not escalate. Escalation is skipped for a non-GET method, response_format=pdf, a screenshot, actions, a proxy that is not http://, or a max_cost below the escalated price; an allowance that covers the first attempt but not the escalation runs the request without it. X-Engine reports the rendering tier that served, each abandoned tier adds an X-Warning: ESCALATED, and diagnostics.attempts lists every attempt (in the JSON envelope on a successful escalation). Batch items never escalate.

Status: the HTTP status is the platform's. A blocked or erroring site is usually still HTTP 200 - read envelope status or X-Target-Status for the site's answer. Errors are application/problem+json; retry only when retryable is true.

Not idempotent: there is no idempotency key, and each call bills again, whether it performs a new scrape or is served from cache. GET and POST accept the same parameters and behave identically.

Authorization

bearerAuth

AuthorizationBearer <token>

Authorization: Bearer <key>. Read the key from the SPICRAWL_API_KEY environment variable; never hard-code or log it. spicrawl_test_… keys can never spend live credits. Scopes: scrape, batch, sessions (granted by default), browser and read (granted deliberately). A missing scope is 403 ERR::AUTH::INSUFFICIENT_SCOPE naming the scope.

In: header

Request Body

application/json

TypeScript Definitions

Use the request body type in TypeScript.

Body of POST /v1/scrape. Unknown fields (including nested ones) are rejected with 400 ERR::REQUEST::INVALID_PARAMETER; a wrong JSON type is 400 naming the field. Max body 1 MiB (413).

url*string

Target URL; required, must be http or https with a host. A private or reserved address fails with 400 ERR::SECURITY::SSRF_BLOCKED at 0 credits.

Formaturi

Lengthlength <= 8192

method?string

HTTP method sent to the target (case-insensitive). There is no request-body parameter, so non-GET methods are sent without a body; only GET/HEAD/OPTIONS are retried on transient failure and only GET results are cached.

Default"GET"

Value in

"GET"
"POST"
"PUT"
"PATCH"
"DELETE"
"HEAD"
"OPTIONS"

js_render?boolean

Run the page's JavaScript before capture (3 credits on datacenter or your own proxy; 25 on managed residential exits, coming soon). Required by every browser-only flag unless stealth or mode=auto is set; cannot be combined with mode=auto. If omitted, the project's stored default applies.

Defaultfalse

stealth?boolean

Coming soon Render with full-browser rendering hardened against bot detection, flat 25 credits at any proxy tier; wins over js_render when both are set. Cannot be combined with mode=auto; returns 503 ERR::ENGINE::UNAVAILABLE where it is not available. If omitted, the project's stored default applies.

Defaultfalse

solve_captcha?boolean

Solve the captcha widgets embedded in the page (Cloudflare Turnstile, reCAPTCHA v2, hCaptcha) before actions run. Selects full-browser rendering, as stealth does. Cannot be combined with mode=auto, which returns 400 ERR::REQUEST::INCOMPATIBLE_FLAGS. Blocking challenge pages are solved on full-browser renders without this flag when the platform has a provider configured. A successful solve adds 5 credits (see x-spicrawl-credits).

Defaultfalse

impersonate?boolean

Fetch tier only: present a current browser TLS/HTTP2 fingerprint, with matching browser headers and Accept-Encoding, to pass passive fingerprint checks. On by default, free, no JavaScript; false turns it off and the fetch uses a non-browser TLS handshake. Ignored on rendered requests. If omitted, the project's stored default applies, and true when the project sets none.

Defaulttrue

mode?"auto"

auto escalates through the rendering tiers (plain fetch, then JavaScript rendering) until one returns a usable page, billing only the rung that succeeded (0 if all fail) while reserving the dearest rung against quota. Incompatible with js_render and stealth. If omitted, the project's stored default applies.

Value in

"auto"

engine?string

Deprecated

Deprecated and ignored; rendering is chosen automatically from the request's flags and the page, and you are billed for the rendering tier actually used (X-Engine). Accepted so existing clients keep working. The response carries X-Warning: ENGINE_IGNORED.

premium_proxy?boolean

Coming soon; not available yet, so bring your own proxy (residential works best). Will use residential exits from the managed pool: 10 credits for a plain fetch, 25 with JavaScript rendering, 32 with full-browser rendering (stealth stays 25). If no residential exit is available the request fails 503 ERR::PROXY::EXHAUSTED rather than falling back; ignored (with X-Warning PROXY_FLAG_IGNORED) when proxy is set. See "Bot challenges" in the operation description. If omitted, the project's stored default applies.

Defaultfalse

proxy_country?string

Coming soon ISO 3166-1 alpha-2 exit country (case-insensitive), within the residential pool. Requires premium_proxy=true (400 ERR::REQUEST::INCOMPATIBLE_FLAGS otherwise) unless proxy is set, in which case it is kept as the render environment hint (browser timezone and locale follow it) while the custom URL stays the exit; an unavailable country is 503 ERR::PROXY::EXHAUSTED, never substituted. A browser render takes its timezone and locale from this country or, when it is omitted, from the country of the pool exit it was assigned. If omitted, the project's stored default applies.

Match^[A-Za-z]{2}$

proxy?string

Your own proxy URL (http, https, socks5 or socks5h; credentials in userinfo). Takes precedence over premium_proxy and adds 0 proxy credits; proxy_country is kept as the render environment hint; loopback hosts are refused with 400 ERR::SECURITY::SSRF_BLOCKED. An unreachable proxy fails 502 ERR::PROXY::UNREACHABLE. A bot challenge from a named vendor is escalated automatically to stronger rendering on the same exit when the proxy is http:// (residential works best); see "Bot challenges" in the operation description.

proxy_verify?boolean

Probe the custom proxy before spending the engine cost so a dead proxy fails fast. Requires proxy (400 ERR::REQUEST::INCOMPATIBLE_FLAGS otherwise).

Defaultfalse

session_id?string

Reuse a session's cookies and storage (created via /v1/sessions); the request runs with the session's own rendering. Checked before any work, at 0 credits and not retryable - an unknown, malformed or other organization's id is 404 ERR::SESSION::NOT_FOUND, a released session 410 ERR::SESSION::RELEASED, an expired one 410 ERR::SESSION::EXPIRED. A successful request increments the session's usage_count, sets last_used_at and slides expires_at. Requests with a session are never cached.

sticky_key?string

Coming soon Any string; requests sharing it reuse the same pool exit. Forces a pool exit even on deployments whose default is direct egress, so it can fail 503 ERR::PROXY::EXHAUSTED where no pool exists.

wait?integer

Milliseconds to wait after load before capture. Browser-only: needs js_render, stealth, or mode=auto, else 400 ERR::REQUEST::INCOMPATIBLE_FLAGS.

Range0 <= value <= 30000

Default0

wait_for?string

CSS selector to wait for before capture. Browser-only. If it never appears the page is returned as it stood with an X-Warning RENDER_DEGRADED; if the render times out waiting it fails 504 ERR::UPSTREAM::TIMEOUT - raise wait_for_timeout or fix the selector.

wait_for_timeout?integer

Milliseconds to wait for wait_for; 0 uses the engine default. Requires wait_for (400 ERR::REQUEST::INCOMPATIBLE_FLAGS otherwise).

Range0 <= value <= 120000

Default0

actions?array<||||||>

Ordered browser workflow (click, fill, scroll, ...), validated before anything is spent. Browser-only and makes the request uncacheable. A failed step ends the request at 0 credits unless that step has on_error=skip.

Itemsitems <= 50

block_resources?array<>

Subresource classes the browser must not load (plural and short aliases accepted, case-insensitive). If omitted, full-browser rendering blocks nothing, so images and fonts load, including for screenshot, pdf, actions and anti-bot escalation. A list blocks exactly the classes named and none (alone) blocks nothing. Any value other than none is browser-only. If omitted, the project's stored default applies.

headless?boolean

Deprecated

Deprecated and ignored. Rendering is chosen automatically. Accepted so existing clients keep working. The response carries X-Warning: ENGINE_IGNORED.

screenshot?boolean

Capture a screenshot; forces the JSON envelope (image base64 under screenshots). Rendered with full-browser rendering - on a deployment without it, 400 ERR::REQUEST::CAPABILITY_UNSUPPORTED at 0 credits. mode=auto cannot satisfy it (its fetch rung has no rasteriser). If the image is not delivered the page is still returned but charged 0.

Defaultfalse

screenshot_fullpage?boolean

Capture the full scrollable page. Requires screenshot=true; mutually exclusive with screenshot_selector.

Defaultfalse

screenshot_selector?string

CSS selector of the element to capture. Requires screenshot=true; mutually exclusive with screenshot_fullpage.

screenshot_format?string

Image format; omitted uses the engine default (png). Not validated by the API - an unrecognised value silently falls back to the engine default.

Value in

"png"
"jpeg"
"jpg"
"webp"

screenshot_quality?integer

Lossy quality for jpeg/webp; 0 uses the engine default. A value above 0 requires screenshot=true.

Range0 <= value <= 100

Default0

custom_headers?

Headers sent to the target, as name -> string value. Hop-by-hop headers (Connection, Keep-Alive, Proxy-Authenticate, Proxy-Authorization, TE, Trailer, Transfer-Encoding, Upgrade, Host, Content-Length) are refused with 400 ERR::REQUEST::INVALID_PARAMETER. If omitted, the project's stored default applies.

network_capture?

Record the XHR/fetch responses the page itself made (often cleaner JSON than the DOM), returned under network in the JSON envelope. Browser-only; no extra credits.

autoparse?boolean

Harvest the page's own structured metadata (JSON-LD, OpenGraph, Twitter, microdata, RDFa, meta, embedded SPA state) into data, no selectors needed. Forces the JSON envelope; no extra credits; refused on a PDF.

Defaultfalse

links?boolean

Return the page's absolute, de-duplicated targets under links. Forces the JSON envelope; no extra credits; refused on a PDF. If omitted, the project's stored default applies.

Defaultfalse

extract?|

Selector map or JSON Schema form. An empty object is 400. Refused on PDF targets (502 ERR::EXTRACT::FAILED).

extract_preset?string

Coming soon Name of a server-side extraction preset. This deployment has no preset store, so any value fails with 400 ERR::EXTRACT::INVALID_RULES (at 0 credits, after the fetch); use extract inline. Mutually exclusive with extract.

ai_extract?|

Exactly one of prompt or schema (both is 400 INCOMPATIBLE_FLAGS, neither is 400 INVALID_PARAMETER). The model reads the page as cleaned markdown; output lands in data. +4 credits.

main_content_only?boolean

true strips nav/footer/aside to the main article; false keeps the whole document. Omitted is on by default for markdown, text and json, and off for html; pass false to keep the whole page. Applies to markdown/text/cleaned output. A request or project value always wins.

include_tags?array<string>

CSS selectors; keep only matching subtrees before extraction and markdown conversion. Applied before exclude_tags and main_content_only.

exclude_tags?array<string>

CSS selectors; remove matching elements, applied after include_tags and before main_content_only.

parse_pdf?boolean

When the target returns a PDF and markdown/text is requested, parse it to text. false refuses it with 502 ERR::EXTRACT::FAILED (0 credits); response_format=html always returns the raw bytes. If omitted, the project's stored default applies.

Defaulttrue

response_format?string

Body shape: html/markdown/text return the document itself, json the envelope (BREAKING CHANGE: an explicit json request now gets the main-content markdown as content and heading-split sections, not the raw HTML), pdf a PDF printed by a real browser (needs full-browser rendering). extract, autoparse, links, ai_extract, network_capture or screenshot override this to the JSON envelope with X-Warning FORMAT_COERCED. A non-HTML target (image, binary) requested as markdown/text fails 502 ERR::EXTRACT::FAILED at 0 credits. If omitted, the project's stored default applies.

Default"html"

Value in

"html"
"markdown"
"text"
"json"
"pdf"

cache?boolean

Serve a stored result younger than cache_ttl (Cache-State: hit). A hit is billed at the same price as the fetch that stored it: the cache saves the fetch (latency and load on the target), not credits. Only billable successes of GET-method requests without session_id or actions are stored. Set false for time-sensitive data; cache=false with cache_ttl>0 is 400 ERR::REQUEST::INCOMPATIBLE_FLAGS.

Defaulttrue

cache_ttl?integer

Maximum acceptable age of a cached result, in seconds. Default and cap are deployment settings (48h by default); larger values are clamped with X-Warning CACHE_TTL_CLAMPED. 0 disables caching for this request.

Range0 <= value

Default172800

max_cost?integer

Credit ceiling; 0 means none. Checked against the dearest reachable outcome (the top rung under mode=auto) before anything runs, failing 400 ERR::LIMIT::MAX_COST_EXCEEDED at 0 credits.

Range0 <= value

Default0

original_status?boolean

On success, use the target's status as this response's HTTP status instead of 200. Leave off unless you need it: it makes a target 404 indistinguishable from an API error by status alone.

Defaultfalse

allowed_status_codes?array<>

Extra target statuses to treat as billable successes (200, 404 and 410 always are). Other statuses return the target's body at 0 credits, and under mode=auto trigger escalation.

Response Body

`application/problem+json`

curl -X POST "https://example.com/v1/scrape" \  -H "Content-Type: application/json" \  -d '{    "url": "https://example.com/blog/launch",    "response_format": "markdown"  }'

{  "url": "https://example.com/product/1",  "final_url": "https://example.com/product/1",  "status": 200,  "content": "<html><head><title>Walnut Desk</title></head><body>...</body></html>",  "headers": {    "content-type": "text/html; charset=utf-8"  },  "truncated": false,  "credits": 1,  "proxy_source": "pool",  "warnings": [],  "data": {    "title": "Walnut Desk",    "price": "$349.00",    "images": [      "https://example.com/img/desk-1.jpg"    ]  }}

Scrape a URL

Authorization

Request Body

Response Body

200application/json

400application/problem+json

401application/problem+json

402application/problem+json

403application/problem+json

404application/problem+json

410application/problem+json

413application/problem+json

429application/problem+json

500application/problem+json

502application/problem+json

503application/problem+json

504application/problem+json

Selector extraction

Rendered with a screenshot

`application/problem+json`

`application/problem+json`

`application/problem+json`

`application/problem+json`

`application/problem+json`

`application/problem+json`

`application/problem+json`

`application/problem+json`

`application/problem+json`

`application/problem+json`

`application/problem+json`

`application/problem+json`