Agent quickstart

Three steps connect your coding agents to Spicrawl: you create an API key, you paste one prompt, and the agent does the rest. This page is written for you and for the agent that reads it. If you are the agent, go to If you are an AI agent.

Get an API key

An agent cannot sign up for you, so this step is yours.

1. Open app.spicrawl.com and sign up or log in.
2. Open Keys and create a key. It looks like spicrawl_live_…; a spicrawl_test_… key works the same and never spends live credits.
3. Copy it. You paste it into the prompt below.

The free plan is 1,000 credits a month. A successful scrape costs credits and a failed request costs 0 (see Credits).

Paste the setup prompt

The prompt needs @spicrawl/cli 0.1.4 or later: init --all, init --api-key and the repairs list in status --json are new in that release. It asks for @latest, so npx fetches the newest release and there is nothing to install first. An older CLI fails the first command with unknown flag: --all (see Troubleshooting).

Replace both <YOUR_API_KEY> with your key, then paste the whole prompt into your coding agent (Claude Code, Cursor, Codex, VS Code with Copilot and so on) in a session that runs on your own computer.

Set up Spicrawl for yourself. My API key is <YOUR_API_KEY>. Only if your shell runs on my computer, run: npx -y @spicrawl/cli@latest init --all --yes --api-key <YOUR_API_KEY> — it finds every coding agent installed here, adds the Spicrawl MCP server and installs the Spicrawl skill. Its JSON output ends with an element whose "kind" is "report", with "connected", "skipped", "api_key" and "next": tell me which agents connected, which were skipped and why, and each step in "next", such as restarting my agents. Check it: npx -y @spicrawl/cli@latest status --json. If a check failed, run the repairs it lists in order; repairs with unattended_safe: false are mine, so show me the command instead of running it. Prove it works: run npx -y @spicrawl/cli@latest scrape https://news.ycombinator.com and tell me today's top story. Report anything you could not finish. Never repeat my API key back to me. If you have no shell on my computer (a chat app or cloud sandbox), stop and send me to https://docs.spicrawl.com/agents/quickstart instead. New MCP tools appear only in a session started after the install.

What the agent does with it:

Three things to know before you paste:

• The key is in your chat history once you paste it. The prompt tells the agent never to repeat it, but do not share the transcript. If you do, delete the key under Keys and create another.
• The key is written in plain text. init --api-key puts it in each agent's own config file in your home directory (~/.claude.json, ~/.cursor/mcp.json, ~/.codex/config.toml, and VS Code's user mcp.json; file mode 0600) and in the CLI's config file. Keep those files out of repositories. The key is also on the command line while init runs, so other users of the same computer can see it in the process list. To avoid both, export the key and leave out --api-key.
• The MCP tools exist only in a session started after the install. The agent proves the key works through the spicrawl CLI or POST /v1/scrape instead. Start a new session to get the spicrawl_* tools.

init --api-key writes the user-level files, which sit outside your repositories. To write the project's own files instead (.mcp.json, .cursor/mcp.json and so on), add --global=false. Those files get the key too, and init adds them to the project's .gitignore when the project has one (a record with kind gitignore). The plain-text-key and .gitignore warnings print in human mode only. In JSON mode, which is what the prompt above runs, stderr stays empty and the mcp record's "secret": true and its reason are the only signal, so an agent that writes project files must check the project's .gitignore itself.

Running the prompt again is safe. init saves the key only after the changes are applied, so a run that is refused leaves the saved key as it was. With a new key it updates its own spicrawl entries in place, keeping the old file as <file>.bak, without --force. An entry you edited by hand (anything but the key) is left alone and the run is refused until you pass --force.

If you are an AI agent

Follow these steps in order. You are done when you have sent the report in step 6.

1. Get the key from the user. You need a Spicrawl API key (spicrawl_live_… or spicrawl_test_…). If the user has not given you one, stop and tell them to sign up at app.spicrawl.com, open Keys and create one. You cannot sign up for them.

2. Check where your shell runs. Run the install only if your shell runs on the user's own computer. In a chat app, a cloud sandbox, a container or a hosted agent (Devin, Codex cloud, Cursor cloud agents), the install would configure the sandbox, not the user's agents. Stop, send the user to https://docs.spicrawl.com/agents/quickstart, and point them to No shell: chat apps and cloud sandboxes.

3. Install. Run the init command from the setup prompt with the user's key. It checks the key first (exit code 3 and nothing written means the API rejected it), writes it into each agent's user-level MCP config and saves it, so nothing needs exporting. stdout is a JSON array with one record per file it touched: client, kind (mcp, skill, agents_md or gitignore), file and action (created, updated, unchanged or skipped, with a reason). Its last element is the report, the one whose kind is report; stderr stays empty:

   {
     "kind": "report",
     "connected": ["claude", "cursor"],
     "skipped": [
       { "client": "vscode", "reason": "not detected" },
       { "client": "codex", "reason": "not detected" }
     ],
     "api_key": "spicrawl_live_ab12…cd34",
     "next": [
       "restart Claude Code, Cursor to load the MCP server and the skill",
       "run npx -y @spicrawl/cli@latest status, then ask your agent to scrape a page (or try: npx -y @spicrawl/cli@latest scrape https://example.com)"
     ]
   }

   connected and skipped hold client ids, api_key is masked, and next is what the user still has to do. Keep it for step 6.

4. Check. Run this and read the result:

   npx -y @spicrawl/cli@latest status --json

   The exit code is 0 when every check passed. 3 means the key is missing or rejected (key_valid: false) and 10 means the API could not be reached. 1 means another check failed: the API is not ready, a worker kind has none ready, or an agent found on this computer has no Spicrawl MCP server entry with a usable key (mcp, a local read of the config files). The full report is on stdout whatever the exit code.

   mcp lists agent client ids: detected (found in the project directory or the home directory), connected (has a Spicrawl MCP server entry that is neither of the next two), needs_env (its entry reads ${SPICRAWL_API_KEY}, which is not set where status runs) and stale (its entry holds a key other than the one in use). repairs is an array with one entry per failed check, in the order to run them, and [] when every check passed. Each entry is {check, command, unattended_safe, why}, where check is reachable, ready, key, workers.<kind> or mcp:

   "mcp": { "detected": ["claude", "cursor"], "connected": ["cursor"], "needs_env": ["claude"], "stale": [] },
   "repairs": [
     {
       "check": "mcp",
       "command": "npx -y @spicrawl/cli@latest init --client claude --yes --global=false",
       "unattended_safe": true,
       "why": "The MCP config for Claude Code reads the key from $SPICRAWL_API_KEY, which is not set here, so the MCP server cannot authenticate; this rewrites the entry with the key saved in the CLI config (the old file is kept as .bak)"
     }
   ]

   If a check failed, run the entries in repairs in the order listed, then run status --json again. Run an entry only when its unattended_safe is true. For false, show the user the command and stop there; it is theirs to run. A command holds the placeholder <YOUR_API_KEY>, never a key. A needs_env or stale repair is init --client <ids> --yes (with --global=false for an entry in the project's own files). A Claude Code needs_env entry is always one, because its user-level config never holds a ${SPICRAWL_API_KEY} reference, as in the example above. The rewrite then puts the key into that project file: its mcp record has "secret": true, so check the project's .gitignore. With a key saved, which the install in step 3 did, it rewrites the entry with that key and is unattended_safe: true: there is no export step. With no key saved, the command carries --api-key <YOUR_API_KEY> and is unattended_safe: false.

5. Prove it. Run npx -y @spicrawl/cli@latest scrape https://news.ycombinator.com and tell the user the top story. The MCP tools are not loaded in your current session, so use the CLI, or POST https://api.spicrawl.com/v1/scrape (see Quickstart).

6. Report. Tell the user which agents connected (connected in the report) and which were skipped and why (skipped), relay each step in next (such as restarting their agents), say anything you could not finish, and tell them they must start a new session in each agent to get the spicrawl_* tools. Do not repeat the key, not even in part: leave api_key out of your message.

What you never do:

• Never repeat the key. Do not echo it in your reply, write it into a file in a repository, put it in a commit or log it. When you show the user a command, replace the key with <YOUR_API_KEY>.
• Never run the install in a cloud sandbox or any shell that is not on the user's computer.
• Never run a repair whose unattended_safe is false.
• Never say the MCP tools are available in the current session. They load when a session starts.
• Never guess a flag. Read the CLI's --help or search the docs with spicrawl_docs_search once the tools are loaded. The docs are also at https://docs.spicrawl.com/llms.txt.

No shell: chat apps and cloud sandboxes

A chat app or a cloud sandbox cannot run the install on your computer. Connect it to the hosted MCP server instead:

https://mcp.spicrawl.com/mcp

Send your key as Authorization: Bearer $SPICRAWL_API_KEY. The server accepts API keys, not OAuth, so a connector that cannot send a custom Authorization header cannot connect. Find where your agent runs:

An agent with an HTTP tool but no MCP support can call the API directly: Quickstart.

Set up with an exported key

This keeps the key out of the agents' config files and out of the process list, as long as no key is saved in the CLI's config file (spicrawl login saves one). Put the key in the SPICRAWL_API_KEY environment variable of the shell your agent starts from, then run init from your project's root directory:

export SPICRAWL_API_KEY=<YOUR_API_KEY>   # add this to your shell profile
npx -y @spicrawl/cli@latest init --yes

init detects the agent tools in use (Claude Code, Cursor, VS Code, Codex) in the project and your home directory, adds the hosted MCP server https://mcp.spicrawl.com/mcp to each client's config and writes the Spicrawl agent skill (SKILL.md). Without --api-key and with no key saved, the config references $SPICRAWL_API_KEY instead of copying the key wherever the client supports it, and init neither checks nor saves the key (spicrawl login does). Once a key is saved, by spicrawl login or an earlier init --api-key, init writes it into the user-level files instead, with no new check; --use-env=true keeps the reference. Without --yes, init shows the files it would create or change and asks first.

To do the two halves separately, or to see what would be written first:

spicrawl mcp install --client cursor --print     # print the MCP config instead of writing it
spicrawl skill install --client claude           # write only the skill

Check it with npx -y @spicrawl/cli@latest status --json, then restart your agent. Every flag and the exact file per client are on Agent setup; install the binary with Install the CLI. On 0.1.3, status has no mcp or repairs and exits 0 whenever the API is reachable and the key works, even if the API is not ready.

Set up by hand with a prompt (no CLI)

If Node.js is not installed, or you do not want to run the CLI, give your agent this prompt. It is self-contained: the agent needs no other page to follow it, and it needs SPICRAWL_API_KEY set in the environment the agent starts from.

Set up Spicrawl (web data API for agents) in this project. Follow these steps exactly.

1. Check that the environment variable SPICRAWL_API_KEY is set (do not print its value).
   If it is not set, stop and ask me to run: export SPICRAWL_API_KEY=spicrawl_live_...
   Never write the key itself into any file in this repository.

2. Add the hosted Spicrawl MCP server for the agent tool you are running in.
   URL: https://mcp.spicrawl.com/mcp (MCP Streamable HTTP)
   Auth header: Authorization: Bearer <SPICRAWL_API_KEY>
   Use the tool's environment-variable syntax so the key stays out of the file:
   - Claude Code: in .mcp.json at the project root, under "mcpServers":
     "spicrawl": {"type": "http", "url": "https://mcp.spicrawl.com/mcp",
                "headers": {"Authorization": "Bearer ${SPICRAWL_API_KEY}"}}
   - Cursor: in .cursor/mcp.json, under "mcpServers":
     "spicrawl": {"url": "https://mcp.spicrawl.com/mcp",
                "headers": {"Authorization": "Bearer ${env:SPICRAWL_API_KEY}"}}
   - VS Code: in .vscode/mcp.json, add to "inputs":
     {"type": "promptString", "id": "spicrawl-api-key", "description": "Spicrawl API key", "password": true}
     and under "servers":
     "spicrawl": {"type": "http", "url": "https://mcp.spicrawl.com/mcp",
                "headers": {"Authorization": "Bearer ${input:spicrawl-api-key}"}}
   - Codex: in ~/.codex/config.toml (or .codex/config.toml in a trusted project):
     [mcp_servers.spicrawl]
     url = "https://mcp.spicrawl.com/mcp"
     bearer_token_env_var = "SPICRAWL_API_KEY"
   Merge into existing files; do not remove other servers.

3. Install the Spicrawl agent skill. Download https://app.spicrawl.com/skill.md and save it as
   SKILL.md in a folder named spicrawl inside the skills directory:
   - Claude Code: .claude/skills/spicrawl/SKILL.md
   - Cursor: .cursor/skills/spicrawl/SKILL.md
   - Codex: .agents/skills/spicrawl/SKILL.md
   - VS Code: .github/skills/spicrawl/SKILL.md

4. Append this section to the project's agent instructions file (CLAUDE.md for Claude Code,
   AGENTS.md otherwise; create it if missing):

   ## Web data (Spicrawl)
   - Fetch web pages with the spicrawl_* MCP tools, or POST https://api.spicrawl.com/v1/scrape
     with Authorization: Bearer $SPICRAWL_API_KEY. Docs: https://docs.spicrawl.com/llms.txt
   - Ask for markdown (response_format "markdown"; the spicrawl_scrape tool's format "markdown").
   - The site's own status is X-Target-Status (or "status" in the JSON envelope), not the
     HTTP status. 200 is the page; 404 or 410 means the page does not exist; anything
     else (403, 429, 503) means the site refused you, so escalate as below.
   - On an error, switch on "code" and retry only when "retryable" is true, after
     retry_after_seconds. Act on diagnostics.hint before retrying.
   - Escalate one step at a time: plain fetch, then js_render, then your own proxy
     if you have one. Set max_cost on every request.
   - For more than 20 URLs use a batch job (spicrawl_batch_submit or POST /v1/batch). Batch
     items come back as html, markdown or text; extraction and screenshots are refused
     with a 400, so scrape those URLs one by one.

5. Verify: run this and confirm the HTTP status is 200 and X-Target-Status is 200:
   curl -sS -D - -o /dev/null -X POST https://api.spicrawl.com/v1/scrape \
     -H "Authorization: Bearer $SPICRAWL_API_KEY" -H "Content-Type: application/json" \
     -d '{"url": "https://example.com", "response_format": "markdown"}'
   A 401 means the key is wrong or revoked. Then tell me which files you changed and that I
   need to restart or reload the agent tool for the MCP server to appear.

Troubleshooting

Next steps

• Per-client details: Claude Code, Cursor, Codex.
• Every MCP tool and its inputs: MCP server.
• The four ways an agent can use Spicrawl: Build with agents.
• Error handling and cost control for agents you build: Best practices.