Quickstart
Get a Spicrawl API key, scrape your first page as markdown with curl, Python, TypeScript or the CLI, and read the response headers that matter.
You need a Spicrawl account and one of: curl, Python 3 with requests, Node.js 18 or later, or the spicrawl CLI. For Node.js there is also a typed client, npm install @spicrawl/sdk. The first scrape below costs 1 credit.
Cloudflare-protected pages are now supported when you bring your own proxy (residential works best); automatic handling on managed premium exits is coming soon Coming soon. See Anti-bot.
Get an API key
Sign in at app.spicrawl.com and open API Keys. Every workspace has a live key named Default that you can reveal and copy, or you can create a new key. A new key's secret is shown once, so copy it now.
Keys look like spicrawl_live_… (spends credits) or spicrawl_test_… (can never spend live credits). See Authentication.
Set SPICRAWL_API_KEY
Every example in these docs, the CLI and the MCP server read the key from this variable.
export SPICRAWL_API_KEY="spicrawl_live_…"Keep the key out of source code and logs. In production, load it from your secret manager.
Scrape a page as markdown
Send the URL and response_format: "markdown". The response body is the page's main content as markdown.
curl -sS -D - "https://api.spicrawl.com/v1/scrape" \
-H "Authorization: Bearer $SPICRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/products/42", "response_format": "markdown"}'Run the TypeScript file with npx tsx quickstart.ts.
Read the headers that matter
A successful call looks like this:
HTTP/2 200
content-type: text/markdown; charset=utf-8
x-request-id: 01M0HF5WFWE7PRE8KZHDTNETWN
x-target-status: 200
x-engine: fetch
x-credits-charged: 1
x-request-cost: 1
x-credits-remaining: 999
cache-state: miss| Header | Check it because |
|---|---|
X-Target-Status | The HTTP 200 is Spicrawl's. This is what the site answered. 403 or 503 here means you got a block page, charged 0; see anti-bot. |
X-Credits-Charged | What this request billed. 0 on any failure or site status other than 200, 404, 410. |
X-Credits-Remaining | Credits left in your monthly allowance after this request. Absent if your organization has no monthly limit. |
X-Engine | The rendering tier that served the page: fetch is a plain fetch with no JavaScript (1 credit), js is JavaScript rendering (3 credits for js_render) and browser is full-browser rendering (8 for a screenshot or PDF). You do not choose it; the flags you send do. |
Cache-State | hit means a stored result was served, billed at the price of the fetch that stored it. miss means it was fetched now. bypass means the request was not cacheable. |
X-Request-Id | Quote it to support, or pass it to GET /v1/requests/{id} for the full trace. |
All headers are listed in Response headers.
If it failed
Errors are application/problem+json with a stable code, a retryable flag and often a diagnostics.hint naming the parameter to change. Failures cost 0 credits.
| Code | Fix |
|---|---|
ERR::AUTH::MISSING_KEY (401) | SPICRAWL_API_KEY is empty in this shell, or the header is missing. |
ERR::REQUEST::INVALID_PARAMETER (400) | A field is misspelled or unknown. detail names it. |
ERR::UPSTREAM::CHALLENGE (502) | The site served a bot challenge. Retry once, then retry through your own residential proxy, where Spicrawl moves to a stronger rendering path by itself. See Anti-bot. |
ERR::UPSTREAM::TIMEOUT (504) | If the page is built by JavaScript, add js_render: true. |
Every code is in Errors.