GitHub Copilot and VS Code

Three steps give GitHub Copilot Spicrawl's tools, the knowledge to write Spicrawl code, and rules for using both well. Copilot runs in several places, and each reads its own MCP config: pick the surface you use in step 1, then do steps 2 and 3 once. Set your key first:

export SPICRAWL_API_KEY=spicrawl_live_...   # add to your shell profile

spicrawl init --client vscode does steps 1 and 2 for VS Code (see What the CLI writes).

1. Add the MCP server

The endpoint is https://mcp.spicrawl.com/mcp (Streamable HTTP) with Authorization: Bearer <your key>. It does not use OAuth, so send the key as a header. See MCP server for each of the 25 spicrawl_* tools.

VS Code agent mode

Create .vscode/mcp.json in the project. VS Code uses the servers key (not mcpServers). An inputs entry with password: true makes VS Code ask for the key once and store it, so the file is safe to commit:

{
  "inputs": [
    {
      "type": "promptString",
      "id": "spicrawl-api-key",
      "description": "Spicrawl API key",
      "password": true
    }
  ],
  "servers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${input:spicrawl-api-key}" }
    }
  }
}

Run MCP: List Servers from the Command Palette and start spicrawl, or use the Start action above the server in the file. Enter the key when prompted. Open the Chat view, switch to Agent mode, and open the tools picker: spicrawl lists 25 spicrawl_* tools.

For every workspace, run MCP: Open User Configuration and put the same inputs and servers there. VS Code also reads a workspace .mcp.json, but that file uses the mcpServers key instead.

From the command line. code --add-mcp adds a server to your user profile. The JSON takes name, the server fields and an optional inputs array (bash or zsh quoting):

code --add-mcp '{"name":"spicrawl","type":"http","url":"https://mcp.spicrawl.com/mcp","headers":{"Authorization":"Bearer ${input:spicrawl-api-key}"},"inputs":[{"type":"promptString","id":"spicrawl-api-key","description":"Spicrawl API key","password":true}]}'

From a link. vscode:mcp/install? followed by the same JSON, URL-encoded, installs the server after a confirmation. Open it in a browser, or with xdg-open on Linux:

vscode:mcp/install?%7B%22name%22%3A%22spicrawl%22%2C%22type%22%3A%22http%22%2C%22url%22%3A%22https%3A%2F%2Fmcp.spicrawl.com%2Fmcp%22%2C%22headers%22%3A%7B%22Authorization%22%3A%22Bearer%20%24%7Binput%3Aspicrawl-api-key%7D%22%7D%2C%22inputs%22%3A%5B%7B%22type%22%3A%22promptString%22%2C%22id%22%3A%22spicrawl-api-key%22%2C%22description%22%3A%22Spicrawl%20API%20key%22%2C%22password%22%3Atrue%7D%5D%7D

Use vscode-insiders: for VS Code Insiders. Keep the key out of the link: the inputs entry makes VS Code ask for it.

Copilot CLI

Add the server with the copilot command. The shell expands $SPICRAWL_API_KEY, so the key is written into ~/.copilot/mcp-config.json:

copilot mcp add --transport http \
  --header "Authorization: Bearer $SPICRAWL_API_KEY" \
  spicrawl https://mcp.spicrawl.com/mcp

Or edit the file yourself:

{
  "mcpServers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer spicrawl_live_..." },
      "tools": ["*"]
    }
  }
}

That file is on your machine only; keep it out of version control. A repository can also hold .mcp.json or .github/mcp.json, but do not commit a key into either. Check the server with copilot mcp list, or /mcp inside a session.

Copilot cloud agent

The cloud agent (the one you assign issues to) runs on GitHub, so it takes its MCP config from the repository, not from your editor. A repository administrator sets it up in two places.

{
  "mcpServers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "tools": ["*"],
      "headers": { "Authorization": "Bearer $COPILOT_MCP_SPICRAWL_API_KEY" }
    }
  }
}

type and tools are required on every server. ["*"] enables all 25 tools; list names such as ["spicrawl_scrape"] to allow fewer. Three things to know:

• The agent calls these tools on its own and does not ask for approval first, and each successful call spends credits. Restrict tools to what the task needs, and use a key with only the scopes you want it to have.
• The cloud agent does not support OAuth servers. That does not affect Spicrawl, which uses a bearer key.
• Only tools are supported. Spicrawl exposes tools only, so nothing is lost.

The agent's firewall does not apply to MCP servers, so mcp.spicrawl.com needs no allowlist entry for the tools to work. It does apply to commands the agent runs in its own shell. If the agent calls the API from code or with curl, add api.spicrawl.com under Settings → Copilot → Internet access → Custom allowlist (a domain entry also covers its subdomains).

2. Install the skill

VS Code, the Copilot CLI and the cloud agent all load skills from .github/skills/, .claude/skills/ and .agents/skills/ in the repository, and from ~/.copilot/skills/ and ~/.agents/skills/ for every project (the cloud agent reads the repository ones):

spicrawl skill install --client vscode

This writes .github/skills/spicrawl/SKILL.md (add --global for ~/.copilot/skills/spicrawl/SKILL.md). Without the CLI:

mkdir -p .github/skills/spicrawl
curl -fsSL https://app.spicrawl.com/skill.md -o .github/skills/spicrawl/SKILL.md

Commit the file so the cloud agent sees it. See Agent skill.

3. Add instructions

Save this as .github/copilot-instructions.md. Copilot in VS Code, the Copilot CLI and the cloud agent all include it in relevant requests, so it applies to every task in the repository. If you already keep this file, paste the section into it.

# Web data (Spicrawl)

Use Spicrawl to read web pages: the spicrawl_* MCP tools in chat, or
POST https://api.spicrawl.com/v1/scrape with `Authorization: Bearer $SPICRAWL_API_KEY` in code.
Docs: https://docs.spicrawl.com/llms.txt (append .md to any page URL for Markdown).

- Ask for markdown: `response_format: "markdown"` (API default is html); on spicrawl_scrape, `format: "markdown"`.
- Check the site's status, not only the HTTP status: `X-Target-Status` header, or `status`
  in the JSON envelope (spicrawl_scrape with `format: "json"`). 200 is the page; 404/410
  mean it does not exist; 403/429/503 mean the site refused, so escalate.
- On an error, switch on `code`. Retry only when `retryable` is true, after
  `retry_after_seconds`. Read `diagnostics.hint` and change what it names first.
  Never retry ERR::REQUEST::*, ERR::AUTH::* or ERR::LIMIT::QUOTA_EXCEEDED.
- Escalate one step at a time and stop at the first that works:
  plain fetch (1 credit) -> `js_render: true` (3) -> add the user's own `proxy`
  if they have one. Empty or skeleton content means render; ERR::UPSTREAM::CHALLENGE
  or a 403 target status means retry once, then the user's own proxy.
- Set `max_cost` on every request to the price of the step you intend.
- Keep the cache on (default). Set `cache: false` only for prices, stock or other live data.
- Trim tokens with `main_content_only` (on by default for markdown), `include_tags`, `exclude_tags`.
- For more than 20 URLs, use a batch job (spicrawl_batch_submit / POST /v1/batch).
  Batch items return raw HTML and apply only render, proxy and block_resources settings.
- Never print or commit SPICRAWL_API_KEY. Log `X-Request-Id` for failures.

Copilot also reads AGENTS.md at the repository root (the nearest one wins). To scope the rules to some files, save them as .github/instructions/spicrawl.instructions.md with frontmatter applyTo: "**/*.ts,**/*.py" (a glob, comma-separated).

What the CLI writes

spicrawl init --client vscode and its two parts, spicrawl mcp install --client vscode and spicrawl skill install --client vscode, target VS Code with Copilot:

It does not write the Copilot CLI or cloud agent config, or the instructions file. Do those by hand as above.

A first task to try

Open Copilot Chat in Agent mode and ask:

Use Spicrawl to read https://example.com/pricing as markdown. List each plan with its monthly price.
If the page comes back empty, retry with rendering.

The agent should call spicrawl_scrape with url and format: "markdown", and add render: true only if the first result is empty. For a coding task:

Add a fetchPage(url) function to this project that calls the Spicrawl API for markdown,
following the Web data instructions. Read the key from process.env.SPICRAWL_API_KEY.

For the cloud agent, open an issue with the same text and assign it to Copilot. Compare the result with the reference implementation in Best practices.

Troubleshooting