Spicrawl
Spicrawl is a web data API for agents: fetch, render and extract any URL as markdown, HTML or structured JSON with one request.
Spicrawl turns any URL into data your code or AI agent can use: clean markdown, HTML, text, a PDF, or JSON extracted with selectors (extraction with a model Coming soon). You send one request shape to https://api.spicrawl.com/v1/scrape, say what you need with flags (js_render, proxy, extract), and Spicrawl picks the engine and retries for you. You pay credits only for successful requests; failures and cache hits cost 0.
curl -sS "https://api.spicrawl.com/v1/scrape" \
-H "Authorization: Bearer $SPICRAWL_API_KEY" \
-H "Content-Type: application/json" \
-d '{"url": "https://example.com/products/42", "response_format": "markdown"}'The response body is the page as markdown. The headers say what the site answered (X-Target-Status), which engine ran (X-Engine) and what it cost (X-Credits-Charged).
Quickstart
Get a key and scrape your first page in curl, Python, TypeScript or the CLI.
Build with agents
Connect Claude Code, Cursor, Codex, GitHub Copilot, OpenCode, Devin and other agents, or your own, through MCP, the skill or the CLI.
CLI
The spicrawl binary: scrape, batch, sessions and logs from a shell or script.
Guides
Render JavaScript, get past anti-bot checks, extract structured data, run batches.
API reference
Every endpoint, parameter and response, generated from the OpenAPI spec.
Errors
Every error code, whether to retry, and which parameter to change.
For AI agents
Agents can use Spicrawl, and read these docs, without a human in the loop:
| Entry point | Where | Use it to |
|---|---|---|
llms.txt | https://docs.spicrawl.com/llms.txt | Load an index of these docs. Append .md to any page URL for its Markdown. See llms.txt. |
| MCP server | https://mcp.spicrawl.com/mcp (streamable HTTP) | Give an MCP client spicrawl_* tools. See MCP. |
| Agent skill | https://app.spicrawl.com/skill.md | Teach a coding agent the HTTP API so it writes correct calls. See Agent skill. |
| CLI | spicrawl init | Detect your agent tooling and install the MCP server and skill in one step. See CLI agent setup. |
| JavaScript/TypeScript SDK | npm install @spicrawl/sdk | Call the API from Node.js 18+ with typed methods instead of raw HTTP. |
Rules an agent should hold on to:
- Two statuses. The HTTP status is Spicrawl's. The site's status is
X-Target-Status. A site that refused you is still HTTP200, at 0 credits. - Switch on
code, retry onretryable. Errors areapplication/problem+json. HonourRetry-After, and readdiagnostics.hint: it names the parameter to change. See Errors. - Failures cost 0.
X-Credits-Chargedis what was billed. - Not idempotent. Retrying a
POST /v1/scrapethat succeeded performs and bills a second scrape. - Unknown fields are rejected. A misspelled parameter is a
400, never ignored.