Set up AI agents with the CLI

Five commands prepare a project for AI coding agents: init does everything, mcp install and skill install do one part each, and docs and schema give an agent reference material from the shell.

export SPICRAWL_API_KEY=spicrawl_live_...
spicrawl init --client claude,cursor --yes

spicrawl init

Detects the agent clients in use (Claude Code, Cursor, VS Code, Codex) in the project directory and your home directory, shows the files it would create or change, and after confirmation runs mcp install and skill install for each.

spicrawl init                                   # detect, show the plan, ask
spicrawl init --client claude,cursor --yes      # no questions
spicrawl init --client all --global --yes       # user-level config for every client
spicrawl init --all --yes --api-key "$SPICRAWL_API_KEY"   # every detected client, key checked, saved and written (0.1.4+)

Without a terminal and without --yes, init writes nothing and exits 2 with init changes files and stdin is not a terminal: re-run with --yes to apply the plan above (in JSON mode the message ends to apply the changes, and the plan is not printed). With no client detected and no --client, it exits 2. A file that is changed is first copied to <file>.bak, and a symlinked config is written through to its target. An existing spicrawl MCP entry that differs from the one to write is not replaced unless you pass --force; the difference is shown and nothing is written or saved. The exception is an entry that differs only in its Authorization: Bearer token (same URL and shape): it is replaced without --force, so a new --api-key works on a re-run.

Output is one record per file touched:

[
  { "client": "claude", "kind": "mcp", "file": "/home/dev/shop/.mcp.json", "action": "created" },
  { "client": "claude", "kind": "skill", "file": "/home/dev/shop/.claude/skills/spicrawl/SKILL.md", "action": "created" }
]

kind is mcp, skill, agents_md or gitignore (see below; its client is empty). action is created, updated, unchanged or skipped (with a reason). A record can also carry backup (the .bak copy) and secret: true (the file holds the literal key). Human mode prints the same as a CLIENT KIND ACTION FILE NOTE table. Re-running is safe.

In JSON mode the array ends with one more element, {"kind": "report", ...}, the summary of the run; see Unattended setup. A consumer that treats every element as a file record must skip it.

Unattended setup: --all and --api-key

One command sets up every detected client from an API key, with no questions asked. It needs @spicrawl/cli 0.1.4 or later (npx -y @spicrawl/cli@latest gets it).

spicrawl init --all --yes --api-key "$SPICRAWL_API_KEY"
spicrawl status --json     # check it; see "Logs, usage and status"

1. --all connects every detected client. It conflicts with --client: both together exit 2 with --all and --client are mutually exclusive.
2. --api-key is checked with the API (GET /v1/requests?limit=1) before anything is planned or written. A rejected key exits 3 with the API's error (ERR::AUTH::INVALID_KEY) and writes and saves nothing. The key is saved to the config file, as spicrawl login does, only after the changes are confirmed (or --yes was given) and applied; the plan lists it as save key. A run that is refused, or declined at the prompt, leaves the saved key as it was.
3. The key is then written into each client's MCP config as is: Authorization: Bearer <key> for Claude Code, Cursor and VS Code, and an http_headers entry for Codex. It goes to the user-level files (~/.claude.json, ~/.cursor/mcp.json, <user config dir>/Code/User/mcp.json, ~/.codex/config.toml), because --global is implied. Each file gets mode 0600. Human mode adds a warning: that it now holds the key in plain text; JSON mode prints no warning, and the file's mcp record carries "secret": true with the reason contains the literal API key. Nothing needs exporting, so the agents authenticate once they are restarted.

Re-running with a new --api-key, or with --api-key after a run that used references, updates the spicrawl entries init wrote earlier in place, with no --force; the old file is kept as <file>.bak. With --global=false, the .bak copy of each project file that holds the key goes into .gitignore as well, because it can hold the previous key. A key file that git already tracks is flagged: a warning in human mode, and tracked by git: git rm --cached <file> in the record's reason.

The key is never printed in full: reports show it masked (spicrawl_live_ab12…cd34), and error output masks it too. The run ends with a report of what connected, what was skipped and what to do next.

In JSON mode, which is what an agent gets when stdout is not a terminal, the report is the last element of the result array and stderr stays empty:

{
  "kind": "report",
  "connected": ["claude", "cursor"],
  "skipped": [
    { "client": "vscode", "reason": "not detected" },
    { "client": "codex", "reason": "not detected" }
  ],
  "api_key": "spicrawl_live_ab12…cd34",
  "next": [
    "restart Claude Code, Cursor to load the MCP server and the skill",
    "run spicrawl status, then ask your agent to scrape a page (or try: spicrawl scrape https://example.com)"
  ]
}

connected and skipped hold client ids; a skipped client has the reason not detected, not selected or the record's own reason. api_key is masked, and "" when no key is configured. next is what the user still has to do. The other elements are unchanged, so a consumer that filters by kind still works.

In human mode the same report prints on stderr, after the result table. On a colour terminal it is a styled Result block (connected, skipped and api key rows) followed by a next list. With NO_COLOR, CLICOLOR=0, CI or TERM=dumb set, or with stderr redirected, it is plain name: value lines:

connected: Claude Code, Cursor
skipped: VS Code (Copilot) (not detected), Codex (not detected)
api key: spicrawl_live_ab12…cd34 (checked, saved to /home/dev/.config/spicrawl/config.json; written into the client configs)
next: restart Claude Code, Cursor to load the MCP server and the skill
next: run spicrawl status, then ask your agent to scrape a page (or try: spicrawl scrape https://example.com)

When the CLI was started by npx, commands in next and in status repairs say npx -y @spicrawl/cli@latest instead of spicrawl. Without a key anywhere, next starts with spicrawl login --api-key <YOUR_API_KEY>, and it includes an export SPICRAWL_API_KEY=<YOUR_API_KEY> step when a client's config reads the variable and it is not exported in the current shell.

The key is a command-line argument, so other users on the computer can see it in the process list while init runs, and it sits in plain text in the client configs. To avoid both, export SPICRAWL_API_KEY and drop --api-key. Claude Code rewrites ~/.claude.json itself: if a running session overwrites the entry, spicrawl status reports the mcp check as failed, and its repair writes the entry again with the saved key (see Repairs). There is no uninstall command; to remove the key, delete the spicrawl entry from each config and revoke the key under Keys.

spicrawl mcp install

Merges a spicrawl server entry for the hosted MCP server (https://mcp.spicrawl.com/mcp, streamable HTTP) into a client's MCP config. Every other server and key in the file is kept. An entry that is already correct is reported as unchanged. A file that is changed is first copied to <file>.bak, and a symlinked config is written through to its target. An existing spicrawl entry that differs from the one to write (say, one you customised) is not replaced unless you pass --force; the difference is shown. VS Code's mcp.json may be JSONC (comments, trailing commas); comments do not survive a rewrite, so a warning is printed and the .bak copy keeps them.

spicrawl mcp install --client claude
spicrawl mcp install --client all --global
spicrawl mcp install --client cursor --print    # show the snippet, write nothing

Flags: --client, --global, --dir, --mcp-url, --use-env, --force as for init, plus --print to print the config snippet instead of writing it.

Per-client result

By default the key is referenced, not copied into the file.

--use-env=false embeds the literal key in every client's file, and so does init --api-key (see Unattended setup; VS Code then has no key prompt, because the header holds the key). Do not commit a file with an embedded key.

{
  "mcpServers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "Bearer ${SPICRAWL_API_KEY}" }
    }
  }
}

With --print in JSON mode the output is [{"client", "file", "format", "snippet"}]; "secret": true marks a snippet that contains the literal key. See MCP server for the tools the server exposes.

spicrawl skill install

Downloads the Spicrawl agent skill from https://docs.spicrawl.com/skill.md (override with --skill-url or $SPICRAWL_SKILL_URL) and writes it as spicrawl/SKILL.md in each client's skill directory. If the download fails, it prints a warning and uses the copy built into the CLI, so it works offline. That copy is taken when the CLI is released, so it can be older than the served skill.

spicrawl skill install --client claude
spicrawl skill install --client all --global
spicrawl skill install --print > SKILL.md

A client that already reads a skill written for another selected client is skipped ("action": "skipped", "reason": "already reads this skill"), so an agent never sees the skill twice. --global writes under your home directory instead of the project. --agents-md also adds a short Spicrawl section to AGENTS.md ($CODEX_HOME/AGENTS.md with --global).

Flags: --client, --global, --dir, --agents-md, --print, --skill-url, --mcp-url. With --print in JSON mode the output is {"content": "...", "source": "<skill URL>" | "embedded"}; source is the URL the skill was downloaded from, or embedded when the download failed and the copy built into the CLI was used. See agent skill.

spicrawl docs

Fetches a page of these docs as Markdown from docs.spicrawl.com and prints it, so an agent can read the docs without a browser. No API key is needed.

spicrawl docs --list                          # llms.txt: every page with its path
spicrawl docs quickstart
spicrawl docs guides/anti-bot > anti-bot.md
spicrawl docs errors --json | jq -r .markdown

A topic is the page's path without the extension (quickstart, guides/anti-bot, cli/scrape). --list, or no topic, prints the site's llms.txt index. Human mode prints the Markdown as is; JSON mode wraps it as {"topic", "url", "markdown"}. An unknown topic exits 2; an unreachable docs host exits 10. Set $SPICRAWL_DOCS_URL to a docs base URL such as https://docs.example.test (a self-hosted API's docs: http://<host>:8080/docs) to read from another host.

spicrawl schema

Prints the JSON Schema (draft 2020-12) of an API request body, so an agent can validate a body before sending it. The schemas are embedded in the CLI: no network, no key.

spicrawl schema --list
spicrawl schema scrape > scrape.schema.json
spicrawl schema batch-item | jq '.properties | keys'

Every $ref is inlined except the recursive extract-rule definition, which lives under $defs in the same document. The output is the schema itself in both human and JSON mode.

# Validate a hand-written body, then send it
spicrawl schema scrape > scrape.schema.json
npx -y ajv-cli validate --spec=draft2020 -s scrape.schema.json -d body.json \
  && spicrawl scrape https://example.com/products/42 --body @body.json