MCP server

The Spicrawl MCP server gives an agent Spicrawl as native tools. It is hosted at:

https://mcp.spicrawl.com/mcp

It speaks MCP Streamable HTTP and authenticates with your own API key, sent as a bearer token or in an x-api-key header. Every tool call runs under that key, exactly as if you had called the API yourself: same scopes, same limits, same credits.

There are two addresses, one per kind of sign-in:

Use the one for your client; the other refuses the wrong kind of sign-in. ChatGPT setup is on ChatGPT.

claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
  --header "Authorization: Bearer $SPICRAWL_API_KEY"

Also on Smithery: smithery.ai/servers/spicrawl/web-scraper (spicrawl/web-scraper). Setup steps are on Smithery.

Authentication

Send your key on every request to the endpoint. The recommended form is Authorization: Bearer spicrawl_live_… (a spicrawl_test_… key works the same). A gateway or client that cannot send a Bearer prefix can use any of these forms; the server takes the first one that holds a well-formed key:

The key is read from headers only. A key in the URL or query string is never read, because URLs end up in access logs; a request that carries it only there counts as having no key.

Without a key

The endpoint answers discovery without a key, so a client or a directory can list the tools before you add one. initialize, ping, the initialized notification and tools/list succeed with HTTP 200. No session is opened and nothing reaches the API. A GET without a key answers 405.

Every other request needs a key, every tools/call included. Without one it is refused with HTTP 401, a WWW-Authenticate: Bearer realm="spicrawl-mcp" header and this message:

Unauthorized: send your Spicrawl API key as 'Authorization: Bearer spicrawl_live_...' (or in an 'x-api-key' header)

With a key

The key is checked against the API when the MCP session opens (initialize):

• A key that is not shaped like a Spicrawl key (spicrawl_live_… or spicrawl_test_…) gets the same 401 as no key. So does a request that names an Mcp-Session-Id without sending the key.
• An unknown, revoked or expired key is refused with HTTP 401, the message Unauthorized: ERR::AUTH::INVALID_KEY: … and the same WWW-Authenticate header, before any session exists.
• If the API cannot be reached to check the key, the endpoint answers 503 with Retry-After: 5. Retry; do not rotate the key.
• A session is bound to the key that opened it. A request that sends another key with that session id is refused with 403.

Keys get the scrape, batch and sessions scopes by default. Two tool groups need a scope you grant deliberately in the dashboard:

Set up your client

Put the key in the SPICRAWL_API_KEY environment variable and reference it from the config, so the key never lands in a committed file.

claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
  --header "Authorization: Bearer $SPICRAWL_API_KEY"

{
  "mcpServers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${SPICRAWL_API_KEY}" }
    }
  }
}

Tools

The server defines 27 tools. The hosted endpoint lists all of them and leaves out every argument marked "Coming soon" (the same as SPICRAWL_MCP_HIDE_UNAVAILABLE, below). Every tool also declares an outputSchema and returns structuredContent; see Structured results. Argument names are the API's own except on spicrawl_scrape and spicrawl_batch_submit, which use render for js_render and format for response_format. Tools reject unknown arguments: a misspelt field (for example js_render on spicrawl_scrape) fails the call with an error naming it, instead of being dropped.

Tool annotations

Every tool declares a title and the three MCP annotations readOnlyHint, destructiveHint and openWorldHint, each as an explicit true or false, so a client can decide which calls to confirm.

Open-world means the tool reaches public or open-ended entities: any URL you give it, or the public docs. The other tools act only on your own Spicrawl account.

spicrawl_scrape is not read-only, although a call with the defaults only reads the page. Its method argument accepts POST, PUT, PATCH and DELETE, and its actions can click, fill and run scripts on the page, so a call can submit a form or write to the target site. Set them only when the task needs it. spicrawl_session_release is destructive because it purges the session's cookies and storage for good.

Scrape

Every other POST /v1/scrape field is accepted under its API name: impersonate, proxy, proxy_verify, method, custom_headers, wait, wait_for_timeout, block_resources, network_capture, screenshot_fullpage, screenshot_selector, screenshot_format, screenshot_quality, parse_pdf, cache_ttl, original_status, allowed_status_codes, extract_preset.

engine and headless are not arguments of spicrawl_scrape, and engine is not an argument of spicrawl_batch_submit or spicrawl_batch_add_items (removed in 0.1.5): rendering is chosen automatically from the other arguments. A call that still sends one fails with an unknown argument error naming it; drop it.

What comes back:

• With format: "markdown", "text" or "html", the tool returns the document itself. Response headers such as X-Target-Status and X-Credits-Charged are not passed through.
• With format: "json", or whenever extract, ai_extract, autoparse, links, network_capture or screenshot is set, the tool returns the JSON envelope: content, the site's status, credits, engine, warnings, empty_fields, and data for extraction. Use it when the agent must check the site's status or the cost.
• Screenshots come back as MCP image blocks the model can see. Each screenshots entry in the JSON keeps its metadata and points to its block. An image over 5 MB of base64 is not attached; its entry says how to get a smaller one.
• format: "pdf" prints the page in a browser (routed to full-browser rendering for you) and returns the file as an MCP resource block (application/pdf). The JSON beside it has engine, status, credits, final_url and pdf.size_bytes. A PDF over 10 MB of base64 is not attached; its pdf.not_attached says to call POST /v1/scrape directly.
• actions is typed for the tool: each step is {"type": "click" | "fill" | "wait_for" | "wait_for_navigation" | "scroll" | "select" | "evaluate" | "screenshot", ...fields}, with the fields of the browser actions verb plus timeout_ms, on_error, label. The tool sends them in the API's {"click": {...}} shape. Infinite scroll: [{"type":"scroll","to_bottom":true},{"type":"wait_for","ms":1000}]. A screenshot step returns the JSON envelope whatever format says.
• Error messages and warnings name the tool's arguments: the API's js_render reads as render, response_format as format.
• ai_extract Coming soon takes prompt or schema, not both, and adds 4 credits. Until AI extraction launches it fails with ERR::INTERNAL::UNAVAILABLE at 0 credits; do not retry.

Batch

max_cost on a batch caps one item; credit_budget caps the whole job. Results expire after the job's retention window (HTTP 410).

Sessions

A session runs one request at a time. A second concurrent scrape on it fails with ERR::SESSION::BUSY (retryable).

Usage and request history

Browser

spicrawl_browser_connect_url returns a connect URL for a remote browser that your own CDP client drives. The browser tools need a key with the browser scope. For scripted page interaction inside one request, use actions on spicrawl_scrape.

While a person has control, the agent's input and navigation commands fail with CDP error -32001; reading the page still works. Poll spicrawl_browser_control until controller is agent, then continue in the same tab with the same cookies. See CDP browser.

Docs

The docs tools read the public docs at https://docs.spicrawl.com and never send your API key there. A self-hosted server reads $SPICRAWL_DOCS_URL, a docs base URL such as http://<host>:8080/docs. When it is unset: the legacy $SPICRAWL_DOCS_HOST plus /docs, then SPICRAWL_PUBLIC_BASE_URL, then SPICRAWL_BASE_URL, plus /docs (a self-hosted API serves its own docs there); with none of them set, or with the public API as the base URL, the public docs site.

Structured results

Since 0.1.7, every tool declares an MCP outputSchema and returns structuredContent, a JSON object, next to the same result as text. A client that supports structured output reads fields such as credits, status or next_cursor directly. Any other client keeps using the text, which is unchanged.

• A result that is a JSON object is the structuredContent as it is; the text is the same object as JSON.
• A result that is a document or an array travels under content, and the text stays the document or array itself. spicrawl_scrape with format set to markdown, text or html returns { "content": "<the document>" }. spicrawl_docs_read and spicrawl_docs_index put the document in content with url and truncated beside it (a page also has title, md_url and section). spicrawl_batch_task_content returns { "content": <payload> } when an item's stored payload is a bare string or a JSON array.
• The schemas are open. Unknown fields are allowed (additionalProperties is true), nearly every field is optional (the few required ones are an id, the list a list tool returns, or content on the docs tools), and values such as a status or an engine tier are plain strings, never enums. A field the API adds later therefore never fails a call.
• A failed call stays an ordinary tool error (isError: true) with no structuredContent; see below.

For example, spicrawl_scrape with extract returns the JSON envelope. Its structuredContent can carry these fields, all optional: content, url, final_url, status, credits, engine, proxy_source, truncated, headers, warnings, data, empty_fields, links, screenshots, network, pdf and diagnostics.

{
  "content": "# Blue widget\n\nPrice: $19.00",
  "url": "https://example.com/products/42",
  "final_url": "https://example.com/products/42",
  "status": 200,
  "credits": 1,
  "engine": "fetch",
  "data": { "title": "Blue widget", "price": "$19.00" }
}

How errors reach the agent

A failed call comes back as an MCP tool error (isError: true) whose text is the API's error code, its detail, a retryable note when retryable is true, and the doc_url:

ERR::UPSTREAM::CHALLENGE: <the API's detail text> (retryable — the same request may succeed on a retry)
See https://docs.spicrawl.com/errors#UPSTREAM_CHALLENGE

The agent should act on the code: retry when the message says retryable, and change parameters otherwise. For ERR::UPSTREAM::CHALLENGE, retry once, then add render: true or the user's own proxy. Failed calls cost 0 credits. The full error table is on Errors.

Self-host over stdio

Run the server as a local subprocess of your MCP client instead of using the hosted endpoint. It is the npm package @spicrawl/mcp (source: github.com/OfficialSpicrawl/mcp), and npx fetches and runs it, so there is nothing to build. It needs Node.js 20 or later.

SPICRAWL_API_KEY=spicrawl_live_... npx -y @spicrawl/mcp

{
  "mcpServers": {
    "spicrawl": {
      "command": "npx",
      "args": ["-y", "@spicrawl/mcp"],
      "env": {
        "SPICRAWL_API_KEY": "spicrawl_live_...",
        "SPICRAWL_BASE_URL": "https://api.spicrawl.com"
      }
    }
  }
}

By default the server lists every tool and argument, including the ones marked Coming soon. With SPICRAWL_MCP_HIDE_UNAVAILABLE=1 it drops every argument marked "Coming soon" from the tool schemas: ai_extract, stealth, extract_preset, premium_proxy, proxy_country and sticky_key on spicrawl_scrape, premium_proxy and proxy_country on the batch tools, and sticky_key, rotate_ip, region_pool, premium_proxy, proxy_country and fingerprint on spicrawl_session_create. A call that still sends one fails with an unknown argument error naming it. Restart the server after changing the variable.

Troubleshooting

Check that the endpoint itself is up. Listing the tools needs no key:

curl -s https://mcp.spicrawl.com/mcp \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

An HTTP 200 with the tool list means the server is reachable; a connection error or a 5xx points at the network or the server, not your client's configuration. (A plain curl -i https://mcp.spicrawl.com/mcp, a GET, answers 405, which also means the server is up.)

Listing does not test your key. To test it, open a session with the key, which spends no credits:

curl -i https://mcp.spicrawl.com/mcp \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json, text/event-stream" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-03-26","capabilities":{},"clientInfo":{"name":"curl","version":"0"}}}'

A valid key answers 200 with an mcp-session-id header. A missing key answers 401 with Unauthorized: send your Spicrawl API key, and an unknown, revoked or expired one 401 with ERR::AUTH::INVALID_KEY.