Claude Code

Set up Spicrawl in Claude Code: add the hosted MCP server, install the agent skill, and add CLAUDE.md rules so the agent fetches web pages cheaply and handles errors correctly.

Three steps give Claude Code Spicrawl's tools, the knowledge to write Spicrawl code, and rules for using both well. Set your key first:

export SPICRAWL_API_KEY=spicrawl_live_...   # add to your shell profile

Or do all three with spicrawl init --client claude.

Or install the Spicrawl plugin, which bundles the MCP server and the skill, from the Spicrawl/agent-plugins marketplace:

claude plugin marketplace add Spicrawl/agent-plugins
claude plugin install spicrawl@spicrawl-plugins

Claude Code asks for your API key when the plugin is enabled and stores it as a sensitive value. With the plugin installed, skip steps 1 and 2.

1. Add the MCP server

For yourself, in this project:

claude mcp add --transport http spicrawl https://mcp.spicrawl.com/mcp \
  --header "Authorization: Bearer $SPICRAWL_API_KEY"

Your shell expands $SPICRAWL_API_KEY when you run the command, and Claude Code stores the header in ~/.claude.json, outside the repository. Add --scope user to have the server in every project.

For the whole team, commit .mcp.json at the project root instead. Claude Code expands ${SPICRAWL_API_KEY} from each person's environment when it starts:

.mcp.json

{
  "mcpServers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${SPICRAWL_API_KEY}" }
    }
  }
}

The same command with --scope project writes this file for you, but with the expanded key in it; edit the header back to ${SPICRAWL_API_KEY} before committing.

Check the connection with claude mcp list, or /mcp inside a session. The server shows 25 spicrawl_* tools. See MCP server for each tool.

2. Install the skill

mkdir -p .claude/skills/spicrawl
curl -fsSL https://app.spicrawl.com/skill.md -o .claude/skills/spicrawl/SKILL.md

Use ~/.claude/skills/spicrawl/ instead to have it in every project. Claude loads the skill when a task involves fetching or extracting web data, or when you type /spicrawl. See Agent skill.

3. Add rules to CLAUDE.md

Paste this section into CLAUDE.md at the project root. Claude Code reads it at the start of every session.

CLAUDE.md

## Web data (Spicrawl)

Use Spicrawl to read web pages: the spicrawl_* MCP tools in chat, or
POST https://api.spicrawl.com/v1/scrape with `Authorization: Bearer $SPICRAWL_API_KEY` in code.
Docs: https://docs.spicrawl.com/llms.txt (append .md to any page URL for Markdown).

- Ask for markdown: `response_format: "markdown"` (API default is html); on spicrawl_scrape, `format: "markdown"`.
- Check the site's status, not only the HTTP status: `X-Target-Status` header, or `status`
  in the JSON envelope (spicrawl_scrape with `format: "json"`). 200 is the page; 404/410
  mean it does not exist; 403/429/503 mean the site refused, so escalate.
- On an error, switch on `code`. Retry only when `retryable` is true, after
  `retry_after_seconds`. Read `diagnostics.hint` and change what it names first.
  Never retry ERR::REQUEST::*, ERR::AUTH::* or ERR::LIMIT::QUOTA_EXCEEDED.
- Escalate one step at a time and stop at the first that works:
  plain fetch (1 credit) -> `js_render: true` (3) -> add the user's own `proxy`
  if they have one. Empty or skeleton content means render; ERR::UPSTREAM::CHALLENGE
  or a 403 target status means retry once, then the user's own proxy.
- Set `max_cost` on every request to the price of the step you intend.
- Keep the cache on (default). Set `cache: false` only for prices, stock or other live data.
- Trim tokens with `main_content_only` (on by default for markdown), `include_tags`, `exclude_tags`.
- For more than 20 URLs, use a batch job (spicrawl_batch_submit / POST /v1/batch).
  Batch items return raw HTML and apply only render, proxy and block_resources settings.
- Never print or commit SPICRAWL_API_KEY. Log `X-Request-Id` for failures.

A first task to try

Start claude in the project and ask:

Use Spicrawl to read https://example.com/pricing as markdown. List each plan with its monthly price.
If the page comes back empty, retry with rendering. Tell me how many credits it cost.

Claude calls spicrawl_scrape. To report the cost it needs format: "json": the envelope carries credits and the site's status, while format: "markdown" returns only the document. For a coding task, try:

Add a function fetch_page(url) to this project that calls the Spicrawl API for markdown,
following the Spicrawl skill's error-handling rules. Read the key from SPICRAWL_API_KEY.

Compare the result with the reference implementation in Best practices.

Troubleshooting

Symptom	Fix
`claude mcp list` shows spicrawl as failed, or 401	The key in the header is wrong, revoked, or the variable was empty. Re-run the `claude mcp add` command in a shell where `echo ${SPICRAWL_API_KEY:0:12}` prints `spicrawl_live_` or `spicrawl_test_`.
`.mcp.json` sends a literal `${SPICRAWL_API_KEY}`	Start `claude` from a shell where the variable is exported.
Claude writes `js_render` on `spicrawl_scrape` and gets "unknown argument"	The tool's argument is `render`. The API field is `js_render`.
Usage tools return `ERR::AUTH::INSUFFICIENT_SCOPE`	Grant the `read` scope to the key in the dashboard.