MCP server

The Spicrawl MCP server gives an agent Spicrawl as native tools. It is hosted at:

https://mcp.spicrawl.com/mcp

It speaks MCP Streamable HTTP and authenticates with your own API key as a bearer token. Every tool call runs under that key, exactly as if you had called the API yourself: same scopes, same limits, same credits.

claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
  --header "Authorization: Bearer $SPICRAWL_API_KEY"

Authentication

Send Authorization: Bearer spicrawl_live_… (or a spicrawl_test_… key) on every request to the endpoint. The key is checked against the API when the MCP session opens:

• An unknown, revoked or expired key is refused with HTTP 401 and a WWW-Authenticate: Bearer realm="spicrawl-mcp" header before any session exists.
• If the API cannot be reached to check the key, the endpoint answers 503 with Retry-After: 5. Retry; do not rotate the key.
• A session is bound to the key that opened it.

Keys get the scrape, batch and sessions scopes by default. Two tool groups need a scope you grant deliberately in the dashboard:

Set up your client

Put the key in the SPICRAWL_API_KEY environment variable and reference it from the config, so the key never lands in a committed file.

claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
  --header "Authorization: Bearer $SPICRAWL_API_KEY"

{
  "mcpServers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${SPICRAWL_API_KEY}" }
    }
  }
}

Tools

The server exposes 25 tools. Argument names are the API's own except on spicrawl_scrape and spicrawl_batch_submit, which use render for js_render and format for response_format. Tools reject unknown arguments: a misspelt field (for example js_render on spicrawl_scrape) fails the call with an error naming it, instead of being dropped.

Scrape

Every other POST /v1/scrape field is accepted under its API name: impersonate, proxy, proxy_verify, method, custom_headers, headless, wait, wait_for_timeout, block_resources, network_capture, screenshot_fullpage, screenshot_selector, screenshot_format, screenshot_quality, parse_pdf, cache_ttl, original_status, allowed_status_codes, extract_preset.

What comes back:

• With format: "markdown", "text" or "html", the tool returns the document itself. Response headers such as X-Target-Status and X-Credits-Charged are not passed through.
• With format: "json", or whenever extract, ai_extract, autoparse, links, network_capture or screenshot is set, the tool returns the JSON envelope: content, the site's status, credits, engine, warnings, empty_fields, and data for extraction. Use it when the agent must check the site's status or the cost.
• Screenshots come back as MCP image blocks the model can see. Each screenshots entry in the JSON keeps its metadata and points to its block. An image over 5 MB of base64 is not attached; its entry says how to get a smaller one.
• format: "pdf" prints the page in a browser (set render or engine: "chromium") and returns the file as an MCP resource block (application/pdf). The JSON beside it has engine, status, credits, request_id and pdf.size_bytes. A PDF over 10 MB of base64 is not attached; its pdf.not_attached says to call POST /v1/scrape directly.
• actions is typed for the tool: each step is {"type": "click" | "fill" | "wait_for" | "wait_for_navigation" | "scroll" | "select" | "evaluate" | "screenshot", ...fields}, with the fields of the browser actions verb plus timeout_ms, on_error, label. The tool sends them in the API's {"click": {...}} shape. Infinite scroll: [{"type":"scroll","to_bottom":true},{"type":"wait_for","ms":1000}]. A screenshot step returns the JSON envelope whatever format says.
• Error messages and warnings name the tool's arguments: the API's js_render reads as render, response_format as format.
• ai_extract Coming soon takes prompt or schema, not both, and adds 4 credits. Until AI extraction launches it fails with ERR::INTERNAL::UNAVAILABLE at 0 credits; do not retry.

Batch

max_cost on a batch caps one item; credit_budget caps the whole job. Results expire after the job's retention window (HTTP 410).

Sessions

A session runs one request at a time. A second concurrent scrape on it fails with ERR::SESSION::BUSY (retryable).

Usage and request history

Browser

Docs

The docs tools read the public docs at https://docs.spicrawl.com and never send your API key there. A self-hosted server reads $SPICRAWL_DOCS_URL, a docs base URL such as http://<host>:8080/docs. When it is unset: the legacy $SPICRAWL_DOCS_HOST plus /docs, then SPICRAWL_PUBLIC_BASE_URL, then SPICRAWL_BASE_URL, plus /docs (a self-hosted API serves its own docs there); with none of them set, or with the public API as the base URL, the public docs site.

How errors reach the agent

A failed call comes back as an MCP tool error (isError: true) whose text is the API's error code, its detail, a retryable note when retryable is true, and the doc_url:

ERR::UPSTREAM::CHALLENGE: <the API's detail text> (retryable — the same request may succeed on a retry)
See https://docs.spicrawl.com/errors#UPSTREAM_CHALLENGE

The agent should act on the code: retry when the message says retryable, and change parameters otherwise. For ERR::UPSTREAM::CHALLENGE, retry once, then add render: true or the user's own proxy. Failed calls cost 0 credits. The full error table is on Errors.

Self-host over stdio

Run the server as a local subprocess of your MCP client instead of using the hosted endpoint. It needs Node.js 20 or later.

cd spicrawl/mcp
npm install
npm run build

{
  "mcpServers": {
    "spicrawl": {
      "command": "node",
      "args": ["/absolute/path/to/spicrawl/mcp/dist/index.js"],
      "env": {
        "SPICRAWL_API_KEY": "spicrawl_live_...",
        "SPICRAWL_BASE_URL": "https://api.spicrawl.com"
      }
    }
  }
}

Troubleshooting

Check that the endpoint itself is up with curl -i https://mcp.spicrawl.com/mcp. Sent without a key, it answers HTTP 401 with Unauthorized: send your Spicrawl API key, which means the server is reachable; a connection error or a 5xx points at the network or the server, not your client's configuration.