Cline

Set up Spicrawl in Cline, the VS Code extension: add the hosted MCP server to cline_mcp_settings.json, install the agent skill, and add a rule so the agent fetches web pages cheaply and handles errors correctly.

Three steps give Cline Spicrawl's tools, the knowledge to write Spicrawl code, and rules for using both well. Cline's documentation describes no environment-variable expansion in its MCP settings, so the key goes into the settings file itself. That file lives in your home directory, outside any repository. Have your key ready:

export SPICRAWL_API_KEY=spicrawl_live_...   # used by code the agent writes; Cline's MCP settings need the literal key

1. Add the MCP server

Open Cline's MCP settings file: click the MCP Servers icon in the top toolbar of the Cline panel, open the Configure tab, then click Configure MCP Servers. Add Spicrawl under mcpServers. Set type to streamableHttp: without it Cline defaults to the legacy sse transport.

cline_mcp_settings.json

{
  "mcpServers": {
    "spicrawl": {
      "type": "streamableHttp",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer spicrawl_live_..." },
      "disabled": false,
      "autoApprove": []
    }
  }
}

Replace spicrawl_live_... with your key. The file is ~/.cline/data/settings/cline_mcp_settings.json. Never commit it or paste it into a shared repository.

Instead of editing JSON, you can use the Remote Servers tab: enter spicrawl as the server name, https://mcp.spicrawl.com/mcp as the URL, choose Streamable HTTP, and click Add Server. That form has no header field, so add the headers object in the JSON afterwards.

The spicrawl server should list 25 spicrawl_* tools. See MCP server for each tool. Keep autoApprove empty at first, or list only read-only tools you trust.

2. Install the skill

Cline loads skills from .cline/skills/, .clinerules/skills/ and .claude/skills/ in the project, and from ~/.cline/skills/ for every project. The skill's directory name must match its name, which is spicrawl:

mkdir -p .cline/skills/spicrawl
curl -fsSL https://app.spicrawl.com/skill.md -o .cline/skills/spicrawl/SKILL.md

Or run spicrawl skill install --client claude, which writes .claude/skills/spicrawl/SKILL.md, a path Cline also reads. Cline shows the skill's description to the model and loads the body when a task matches. You can also type / in the chat to run it. Manage skills from the scale icon at the bottom of the Cline panel, on the Skills tab. See Agent skill.

3. Add rules

Cline reads workspace rules from .clinerules/ (a folder of Markdown files) or .cline/rules/, and it reads AGENTS.md at the project root. Save this as .clinerules/spicrawl.md:

.clinerules/spicrawl.md

# Web data (Spicrawl)

Use Spicrawl to read web pages: the spicrawl_* MCP tools in chat, or
POST https://api.spicrawl.com/v1/scrape with `Authorization: Bearer $SPICRAWL_API_KEY` in code.
Docs: https://docs.spicrawl.com/llms.txt (append .md to any page URL for Markdown).

- Ask for markdown: `response_format: "markdown"` (API default is html); on spicrawl_scrape, `format: "markdown"`.
- Check the site's status, not only the HTTP status: `X-Target-Status` header, or `status`
  in the JSON envelope (spicrawl_scrape with `format: "json"`). 200 is the page; 404/410
  mean it does not exist; 403/429/503 mean the site refused, so escalate.
- On an error, switch on `code`. Retry only when `retryable` is true, after
  `retry_after_seconds`. Read `diagnostics.hint` and change what it names first.
  Never retry ERR::REQUEST::*, ERR::AUTH::* or ERR::LIMIT::QUOTA_EXCEEDED.
- Escalate one step at a time and stop at the first that works:
  plain fetch (1 credit) -> `js_render: true` (3) -> add the user's own `proxy`
  if they have one. Empty or skeleton content means render; ERR::UPSTREAM::CHALLENGE
  or a 403 target status means retry once, then the user's own proxy.
- Set `max_cost` on every request to the price of the step you intend.
- Keep the cache on (default). Set `cache: false` only for prices, stock or other live data.
- Trim tokens with `main_content_only` (on by default for markdown), `include_tags`, `exclude_tags`.
- For more than 20 URLs, use a batch job (spicrawl_batch_submit / POST /v1/batch).
  Batch items return raw HTML and apply only render, proxy and block_resources settings.
- Never print or commit SPICRAWL_API_KEY. Log `X-Request-Id` for failures.

If you already keep agent instructions in AGENTS.md, paste the same section into it instead. Keep each rule file to one concern so you can toggle it on its own.

A first task to try

Open a Cline task and ask:

Use Spicrawl to read https://example.com/pricing as markdown. List each plan with its monthly price.
If the page comes back empty, retry with rendering.

The agent should call spicrawl_scrape with url and format: "markdown", and add render: true only if the first result is empty. Cline asks you to approve the tool call unless it is in autoApprove. For a coding task:

Add a fetchPage(url) function to this project that calls the Spicrawl API for markdown,
following the Web data rules in .clinerules. Read the key from process.env.SPICRAWL_API_KEY.

Compare the result with the reference implementation in Best practices.

Troubleshooting

Symptom	Fix
The server fails to connect or shows a transport error	`type` is missing, so Cline used SSE. Set `"type": "streamableHttp"` and restart the server from the MCP Servers panel.
401 in the server's error log	The `Authorization` header is empty or wrong. Do not write `$SPICRAWL_API_KEY` in this file: paste the literal key after `Bearer` .
The server is listed but has no tools	Restart it from the MCP Servers panel, then start a new task.
The agent writes `js_render` on `spicrawl_scrape` and gets "unknown argument"	The tool's argument is `render`. The API field is `js_render`.
Usage tools return `ERR::AUTH::INSUFFICIENT_SCOPE`	Grant the `read` scope to the key in the dashboard.