# JetBrains AI Assistant and Junie

> Set up Spicrawl in JetBrains IDEs: add the hosted MCP server to AI Assistant and Junie, install the agent skill, and add project instructions so the agent fetches web pages cheaply and handles errors correctly.

Source: https://docs.spicrawl.com/agents/jetbrains

JetBrains IDEs have two agents that connect to MCP servers: AI Assistant (its own settings page and its own rules) and Junie (its own `mcp.json`, skills and guidelines). Both talk to Streamable HTTP servers. The Spicrawl CLI has no JetBrains target (`--client` accepts `claude`, `cursor`, `vscode`, `codex` and `all`), so you do each step by hand. Have your key ready:

```bash
export SPICRAWL_API_KEY=spicrawl_live_...   # for your own code and the skill's examples
```

## 1. Add the MCP server

Both agents use the same server entry: the hosted endpoint plus your key as a bearer header. Spicrawl has no OAuth, so the header is required.

**AI Assistant**

Open **Settings | Tools | AI Assistant | Model Context Protocol (MCP)** and click **Add**. JetBrains documents only a `url` for HTTP servers, not request headers, so connect through the [`mcp-remote`](https://www.npmjs.com/package/mcp-remote) bridge (needs Node.js) on the **STDIO** transport. Paste this JSON:

```json
{
  "mcpServers": {
    "spicrawl": {
      "command": "npx",
      "args": [
        "-y", "mcp-remote", "https://mcp.spicrawl.com/mcp",
        "--header", "Authorization:${SPICRAWL_AUTH}"
      ],
      "env": { "SPICRAWL_AUTH": "Bearer spicrawl_live_..." }
    }
  }
}
```

The header value goes through the `SPICRAWL_AUTH` variable because some clients split arguments on spaces. Click **OK**, then **Apply**. The server appears in the list with its connection status and tools; it should show 25 `spicrawl_*` tools. Set the level to **Global** to use it in every project, or **Project** to scope it. The key is stored in the IDE's MCP settings, not in your repository.

If your AI Assistant version accepts headers on the **HTTP** transport, the direct form is `{"mcpServers": {"spicrawl": {"url": "https://mcp.spicrawl.com/mcp", "headers": {"Authorization": "Bearer spicrawl_live_..."}}}}`. If the server then fails with `401`, use the bridge above.

**Junie**

Junie reads `mcp.json` at two scopes. The project file can be shared through version control, so keep the key out of it; the user file is private to you:

| Scope   | File                    |
| ------- | ----------------------- |
| User    | `~/.junie/mcp/mcp.json` |
| Project | `.junie/mcp/mcp.json`   |

```json title="~/.junie/mcp/mcp.json"
{
  "mcpServers": {
    "spicrawl": {
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer spicrawl_live_..." }
    }
  }
}
```

The Junie plugin's MCP settings list each server with its status and where it is stored (global or project). In the Junie CLI, run `/mcp` to see each server's status (Active, Inactive, Failed, Disabled), and to enable, disable or edit it.

See [MCP server](https://docs.spicrawl.com/agents/mcp.md) for each tool.

## 2. Install the skill

Junie loads skills from `.junie/skills/` in the project and `~/.junie/skills/` for every project, and also from `.agents/skills/`. Each skill is a folder with a `SKILL.md`:

```bash
mkdir -p .agents/skills/spicrawl
curl -fsSL https://app.spicrawl.com/skill.md -o .agents/skills/spicrawl/SKILL.md
```

`spicrawl skill install --client codex` writes the same file. Junie selects the skill when a task matches its `description`; run `/skills` to see what it discovered, or `/spicrawl` to apply it directly. See [Agent skill](https://docs.spicrawl.com/agents/skill.md).

AI Assistant does not read this folder; it relies on the rules in step 3.

## 3. Add instructions

**Junie**

Junie reads `AGENTS.md` at the project root (with `.junie/playbook.md` and `.junie/rules/*.md` if you have them), or `.junie/AGENTS.md`, and the older `.junie/guidelines.md`. Use `AGENTS.md`, which other agents read too. `~/.junie/AGENTS.md` holds global guidelines; project guidelines win on conflict.

```markdown title="AGENTS.md"
## Web data (Spicrawl)

Use Spicrawl to read web pages: the spicrawl_* MCP tools in chat, or
POST https://api.spicrawl.com/v1/scrape with `Authorization: Bearer $SPICRAWL_API_KEY` in code.
Docs: https://docs.spicrawl.com/llms.txt (append .md to any page URL for Markdown).

- Ask for markdown: `response_format: "markdown"` (API default is html); on spicrawl_scrape, `format: "markdown"`.
- Check the site's status, not only the HTTP status: `X-Target-Status` header, or `status`
  in the JSON envelope (spicrawl_scrape with `format: "json"`). 200 is the page; 404/410
  mean it does not exist; 403/429/503 mean the site refused, so escalate.
- On an error, switch on `code`. Retry only when `retryable` is true, after
  `retry_after_seconds`. Read `diagnostics.hint` and change what it names first.
  Never retry ERR::REQUEST::*, ERR::AUTH::* or ERR::LIMIT::QUOTA_EXCEEDED.
- Escalate one step at a time and stop at the first that works:
  plain fetch (1 credit) -> `js_render: true` (3) -> add the user's own `proxy`
  if they have one. Empty or skeleton content means render; ERR::UPSTREAM::CHALLENGE
  or a 403 target status means retry once, then the user's own proxy.
- Set `max_cost` on every request to the price of the step you intend.
- Keep the cache on (default). Set `cache: false` only for prices, stock or other live data.
- Trim tokens with `main_content_only` (on by default for markdown), `include_tags`, `exclude_tags`.
- For more than 20 URLs, use a batch job (spicrawl_batch_submit / POST /v1/batch).
  Batch items return raw HTML and apply only render, proxy and block_resources settings.
- Never print or commit SPICRAWL_API_KEY. Log `X-Request-Id` for failures.
```

**AI Assistant**

AI Assistant reads project rules from Markdown files in `.aiassistant/rules/`. Create `.aiassistant/rules/spicrawl.md` with the same text, then set its type in AI Assistant's rules settings: **By model decision** (with the instruction "Fetching, scraping or extracting data from web pages") includes it when a task involves web data, and **Always** includes it in every chat. Its docs do not mention `AGENTS.md` for this folder.

## A first task to try

Open the agent chat and ask:

```text
Use Spicrawl to read https://example.com/pricing as markdown. List each plan with its monthly price.
If the page comes back empty, retry with rendering.
```

The agent should call `spicrawl_scrape` with `url` and `format: "markdown"`, and add `render: true` only if the first result is empty. For a coding task:

```text
Add a fetchPage(url) function to this project that calls the Spicrawl API for markdown,
following the Web data instructions. Read the key from process.env.SPICRAWL_API_KEY.
```

Compare the result with the reference implementation in [Best practices](https://docs.spicrawl.com/agents/best-practices.md).

## Troubleshooting

| Symptom                                                                                            | Fix                                                                                                     |
| -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------- |
| The server shows as failed, or the endpoint answers 401 `Unauthorized: send your Spicrawl API key` | The `Authorization` header is missing, wrong, or has no `Bearer ` prefix. Fix the JSON and apply again. |
| The server connects but shows no tools                                                             | Open a new chat, and check the server's tools on the MCP settings page (or `/mcp` in Junie CLI).        |
| A rule in `.aiassistant/rules/` is ignored                                                         | Its type is **Off** or **Manually**. Set it to **Always** or **By model decision**.                     |
| The agent writes `js_render` on `spicrawl_scrape` and gets "unknown argument"                      | The tool's argument is `render`. The API field is `js_render`.                                          |
| Usage tools return `ERR::AUTH::INSUFFICIENT_SCOPE`                                                 | Grant the `read` scope to the key in the dashboard.                                                     |