Agent quickstart

export SPICRAWL_API_KEY=spicrawl_live_...   # add this to your shell profile

spicrawl init

spicrawl init --client claude --yes

spicrawl mcp install --client cursor --print     # print the MCP config instead of writing it
spicrawl skill install --client claude           # write only the skill

Set up Spicrawl (web data API for agents) in this project. Follow these steps exactly.

1. Check that the environment variable SPICRAWL_API_KEY is set (do not print its value).
   If it is not set, stop and ask me to run: export SPICRAWL_API_KEY=spicrawl_live_...
   Never write the key itself into any file in this repository.

2. Add the hosted Spicrawl MCP server for the agent tool you are running in.
   URL: https://mcp.spicrawl.com/mcp (MCP Streamable HTTP)
   Auth header: Authorization: Bearer <SPICRAWL_API_KEY>
   Use the tool's environment-variable syntax so the key stays out of the file:
   - Claude Code: in .mcp.json at the project root, under "mcpServers":
     "spicrawl": {"type": "http", "url": "https://mcp.spicrawl.com/mcp",
                "headers": {"Authorization": "Bearer ${SPICRAWL_API_KEY}"}}
   - Cursor: in .cursor/mcp.json, under "mcpServers":
     "spicrawl": {"url": "https://mcp.spicrawl.com/mcp",
                "headers": {"Authorization": "Bearer ${env:SPICRAWL_API_KEY}"}}
   - VS Code: in .vscode/mcp.json, add to "inputs":
     {"type": "promptString", "id": "spicrawl-api-key", "description": "Spicrawl API key", "password": true}
     and under "servers":
     "spicrawl": {"type": "http", "url": "https://mcp.spicrawl.com/mcp",
                "headers": {"Authorization": "Bearer ${input:spicrawl-api-key}"}}
   - Codex: in ~/.codex/config.toml (or .codex/config.toml in a trusted project):
     [mcp_servers.spicrawl]
     url = "https://mcp.spicrawl.com/mcp"
     bearer_token_env_var = "SPICRAWL_API_KEY"
   Merge into existing files; do not remove other servers.

3. Install the Spicrawl agent skill. Download https://app.spicrawl.com/skill.md and save it as
   SKILL.md in a folder named spicrawl inside the skills directory:
   - Claude Code: .claude/skills/spicrawl/SKILL.md
   - Cursor: .cursor/skills/spicrawl/SKILL.md
   - Codex: .agents/skills/spicrawl/SKILL.md
   - VS Code: .github/skills/spicrawl/SKILL.md

4. Append this section to the project's agent instructions file (CLAUDE.md for Claude Code,
   AGENTS.md otherwise; create it if missing):

   ## Web data (Spicrawl)
   - Fetch web pages with the spicrawl_* MCP tools, or POST https://api.spicrawl.com/v1/scrape
     with Authorization: Bearer $SPICRAWL_API_KEY. Docs: https://docs.spicrawl.com/llms.txt
   - Ask for markdown (response_format "markdown"; the spicrawl_scrape tool's format "markdown").
   - The site's own status is X-Target-Status (or "status" in the JSON envelope), not the
     HTTP status. 200 is the page; 404 or 410 means the page does not exist; anything
     else (403, 429, 503) means the site refused you, so escalate as below.
   - On an error, switch on "code" and retry only when "retryable" is true, after
     retry_after_seconds. Act on diagnostics.hint before retrying.
   - Escalate one step at a time: plain fetch, then js_render, then your own proxy
     if you have one. Set max_cost on every request.
   - For more than 20 URLs use a batch job (spicrawl_batch_submit or POST /v1/batch). Batch
     items come back as the raw page (HTML), not markdown.

5. Verify: run this and confirm the HTTP status is 200 and X-Target-Status is 200:
   curl -sS -D - -o /dev/null -X POST https://api.spicrawl.com/v1/scrape \
     -H "Authorization: Bearer $SPICRAWL_API_KEY" -H "Content-Type: application/json" \
     -d '{"url": "https://example.com", "response_format": "markdown"}'
   A 401 means the key is wrong or revoked. Then tell me which files you changed and that I
   need to restart or reload the agent tool for the MCP server to appear.

Use Spicrawl to read https://example.com/pricing as markdown and list the plans with their prices.