Response headers
Every header Spicrawl returns on /v1/scrape: request id, credits, rendering tier, target status, proxy, cache state, limits and X-Warning codes.
A /v1/scrape response tells you what ran, what it cost and what the site said, in headers, whatever the body format. Read them with curl -D - (dump headers) or from your HTTP client's response object.
curl -sS -D - -o page.md "https://api.spicrawl.com/v1/scrape" \
-H "Authorization: Bearer $SPICRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/products/42", "response_format": "markdown"}'HTTP/2 200
content-type: text/markdown; charset=utf-8
x-request-id: 01M0HF5WFWE7PRE8KZHDTNETWN
x-credits-charged: 1
x-request-cost: 1
x-credits-remaining: 987
x-engine: fetch
x-proxy-source: direct
x-target-status: 200
x-final-url: https://example.com/products/42
x-ratelimit-limit: 10
x-ratelimit-remaining: 9
concurrency-limit: 5
concurrency-remaining: 4
cache-state: missTwo statuses
The HTTP status line is the platform's. The site's status is X-Target-Status.
| HTTP status | X-Target-Status | Meaning | Credits |
|---|---|---|---|
200 | 200 | The site returned the page. | charged |
200 | 404 or 410 | The site said the page does not exist. The call succeeded. | charged |
200 | 403, 500, … | The site refused or failed. You get its body. The call still succeeded. | 0 |
4xx/5xx with application/problem+json | absent | Spicrawl could not complete the request. Read code. | 0 |
Always check X-Target-Status (or envelope status) before trusting the body. A 200 with X-Target-Status: 403 is a block page, not your content. original_status=true moves the target's status onto the HTTP status line; avoid it unless a client requires it, because a target 404 then looks like an API error.
Header reference
| Header | Values | Meaning |
|---|---|---|
X-Request-Id | ULID | This request's id, on every response including errors. Quote it to support; pass it to GET /v1/requests/{id} for the trace. |
X-Credits-Charged | integer | Credits actually billed. 0 on any failure or non-billable target status. On a cache hit, the price of the fetch that stored it. |
X-Request-Cost | integer | Price of the rendering tier and proxy class that ran. On a cache hit, the price of the one that fetched the stored copy. Differs from X-Credits-Charged exactly when nothing was billed. |
X-Credits-Remaining | integer | Credits left in your organization's monthly allowance after this request's charge, rounded down. Counts credits held by running batch jobs and open browser sessions. Never negative. On a 402 it can be above 0: less than the request needed. Absent when your organization has no monthly limit, when the request was refused before the credit check (a validation error, a rate or concurrency limit), or when the balance could not be read. See Credits. |
X-Engine | fetch, js, browser | The rendering tier that served the request, or the last one attempted. Informational, not a setting: fetch is a plain fetch with no JavaScript, js is JavaScript rendering, browser is full-browser rendering. The JSON engine field, diagnostics, request history and sessions use the same three values. Under mode=auto or an anti-bot escalation, the step that produced the result. |
X-Proxy-Source | custom, direct, pool | Whose exit carried the request: your proxy, no proxy, or the managed pool Coming soon. |
X-Proxy-Endpoint | string | Identifier of the managed pool exit that served (managed pool Coming soon). Absent for custom and direct. |
X-Target-Status | integer | The site's HTTP status. Absent when no document was retrieved. |
X-Final-Url | URL | URL after redirects, when known. |
X-RateLimit-Limit | integer | Size of the key's rate-limit bucket: the most requests you can send back to back. |
X-RateLimit-Remaining | integer | Requests left in that bucket right now. Never above X-RateLimit-Limit; it refills continuously. |
X-RateLimit-Reset | integer | Seconds until the bucket has one request available again. Sent only when the limiter imposed a wait. |
Concurrency-Limit | integer | Concurrent in-flight requests allowed for your organization. |
Concurrency-Remaining | integer | Concurrent slots left. |
Cache-State | hit, miss, bypass | Result-cache outcome. Always present on a completed scrape. |
X-Warning | CODE: message | A non-fatal decision the platform made for you. Repeated, one header per warning. |
Retry-After | integer | On retryable errors: seconds to wait before retrying. |
Content-Type | see below | Shape of the body. |
Content-Type follows response_format: text/html (html, the default), text/markdown (markdown), text/plain (text), application/json (json, and any request that forces the envelope), application/pdf (pdf). Errors are always application/problem+json.
Cache-State
| Value | Meaning | Credits |
|---|---|---|
hit | Served from a stored result younger than cache_ttl. Nothing was fetched, but the hit is billed. | same price as the fetch that stored it |
miss | The request was eligible for the cache, nothing fresh was stored, so it was fetched now. A billable success is stored for next time. | normal price |
bypass | The request was not eligible: cache=false or cache_ttl=0, a session_id, actions, a method other than GET, or caching is off on this deployment. | normal price |
The default cache_ttl is 172800 seconds (48 hours). Set cache: false for time-sensitive data. See Caching.
X-Warning codes
Each X-Warning header is CODE: message. Switch on the code; the message is for humans. The JSON envelope's warnings array carries the messages only.
| Code | When you see it | What to do |
|---|---|---|
PROXY_FLAG_IGNORED | You set proxy together with a managed-pool flag (premium_proxy or proxy_country Coming soon). Your own proxy wins and the pool flags were ignored. | Remove the ignored flags. |
FORMAT_COERCED | You asked for html, markdown or text, but extract, autoparse, links, ai_extract, network_capture or screenshot needs the JSON envelope, so response_format became json. | Send response_format: "json" explicitly, or drop the flag. Markdown is in the envelope's content. |
CACHE_TTL_CLAMPED | cache_ttl was above the deployment's maximum (48 hours by default) and was lowered to it. | Send a smaller cache_ttl. |
ENGINE_IGNORED | You sent engine or headless. Both are deprecated and ignored: rendering is chosen automatically from the request. They are accepted so existing clients keep working, and the message says the field was ignored (engine is chosen automatically; the engine field is ignored., or the browser mode is chosen automatically; that field is ignored. for headless). | Remove the field. Send the flag that states what you need (js_render, screenshot, actions, mode=auto). |
ENGINE_SUBSTITUTED | The request was served on a different rendering tier than its flags first select, because the deployment could not serve the usual one. The message names it (ENGINE_SUBSTITUTED: the requested rendering was unavailable; served by JavaScript rendering., or browser rendering). X-Engine names the tier that ran, and X-Request-Cost its price. | Read X-Request-Cost. You cannot choose the tier to prevent it. |
RENDER_DEGRADED | The browser returned the page but could not deliver something you asked for, for example a wait_for selector that never appeared. | Check the selector, or raise wait_for_timeout. |
EMPTY_RENDER | The browser rendered the page but found no visible content, so nothing was charged. The body (if any) is still returned with a 200. | Retry with a wait_for selector that appears when the page is ready, or a longer wait. |
ESCALATED | A cheaper step was tried and abandoned before the one that answered: under mode=auto, or when a bot challenge (Cloudflare and similar) on your own proxy moved the request to a stronger rendering path. One header per abandoned step, sent on success too, so you can see why the price rose. Each names the abandoned tier, that tier's price and the reason: ESCALATED: fetch (1 credit): blocked by challenge page (Cloudflare …), then ESCALATED: JavaScript rendering (3 credits): …. | Under mode=auto, when js_render=true would have worked, send it next time to skip the failed step. |
Warnings also appear in an error body's warnings array when a request fails.
Read the headers in code
curl -sS -D headers.txt -o page.md "https://api.spicrawl.com/v1/scrape" \
-H "Authorization: Bearer $SPICRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/products/42", "response_format": "markdown"}'
grep -i -E '^(x-target-status|x-credits-charged|x-engine|cache-state|x-warning):' headers.txt