Agent skill
Install the Spicrawl agent skill (SKILL.md from https://app.spicrawl.com/skill.md) so a coding agent writes correct calls to the Spicrawl HTTP API.
The Spicrawl agent skill is one Markdown file, SKILL.md, that teaches an AI agent to call the Spicrawl HTTP API: base URL, authentication, request fields, the response shape, error codes and a cost playbook. Agents that support skills load it automatically when a task involves fetching or extracting web data.
It is served at:
https://app.spicrawl.com/skill.mdInstall it with the CLI:
spicrawl skill installSkill or MCP server?
| Skill | MCP server | |
|---|---|---|
| What it gives the agent | Knowledge: how to write HTTP calls to Spicrawl | Tools: the agent calls Spicrawl directly |
| Pick it when | The agent writes code that uses Spicrawl (a scraper in your backend, a pipeline, a script) | The agent fetches pages for you during the conversation |
| Needs at runtime | Nothing; the code it writes needs SPICRAWL_API_KEY | A connection to https://mcp.spicrawl.com/mcp with your key |
Install both if you do both. They do not conflict.
Install
spicrawl skill install # every agent tool detected in the project
spicrawl skill install --client claude # only Claude Code
spicrawl skill install --client all --global # your home directory, every client
spicrawl skill install --print > SKILL.md # print the skill instead of writing itThe CLI downloads https://app.spicrawl.com/skill.md and writes it as spicrawl/SKILL.md in each client's skill directory. If the download fails, it uses a copy built into the CLI and warns on stderr. That copy is taken when the CLI is released, so it can be older than the served skill.
| Client | Project path |
|---|---|
| Claude Code | .claude/skills/spicrawl/SKILL.md |
| Cursor | .cursor/skills/spicrawl/SKILL.md |
| VS Code | .github/skills/spicrawl/SKILL.md (~/.copilot/skills with --global) |
| Codex | .agents/skills/spicrawl/SKILL.md |
A client that already reads a skill written for another selected client is skipped, so an agent never sees the skill twice. --agents-md also adds a short Spicrawl section to AGENTS.md. spicrawl init installs the skill and the MCP server together. See Agent setup.
Use the folder name spicrawl so that spicrawl skill install later overwrites the same file. The file contains no secrets, so it is safe to commit; the code the agent writes reads the key from SPICRAWL_API_KEY.
What it contains
The file starts with the frontmatter agents use to decide when to load it:
---
name: spicrawl-web-data-api
description: Fetch, render and extract data from web pages with the Spicrawl API. Use when an agent needs a page's content as markdown, HTML, text or JSON, has to render JavaScript, get past anti-bot protection, extract structured fields, scrape many URLs as a batch job, keep a logged-in session, or drive a real browser over CDP.
---The body covers:
| Section | What the agent learns |
|---|---|
| Authentication | Authorization: Bearer $SPICRAWL_API_KEY, key formats, the five scopes (scrape, batch, sessions by default; browser, read on request) |
| Scrape one page | POST /v1/scrape in curl and Python, and the request fields with when to use each |
| Structured extraction | Selector maps and JSON Schema extract, autoparse, ai_extract Coming soon |
| Browser actions | The actions array for clicking, filling and scrolling |
| Response | Document body vs. JSON envelope, the two statuses, and the response headers (X-Request-Id, X-Credits-Charged, X-Target-Status, Cache-State, X-Warning) |
| Many URLs | POST /v1/batch, polling, reading JSONL results |
| Sessions and live browser | /v1/sessions, and the CDP WebSocket Coming soon |
| Credits | The price table, and that failures and cache hits cost 0 |
| Errors and retries | Trust retryable, which codes to retry, which to fix |
| Playbook | Start cheap, escalate to js_render, set max_cost, batch for many URLs |
Keep it updated
The skill changes when the API gains fields. The copy you saved does not update itself.
-
Re-run
spicrawl skill install(or thecurlcommand above) after upgrading, or on a schedule in CI. It overwritesspicrawl/SKILL.mdin place. -
To check whether your copy is current, compare it with the served version:
curl -fsSL https://app.spicrawl.com/skill.md | diff - .claude/skills/spicrawl/SKILL.md -
For always-current reference material, point the agent at the docs as well:
https://docs.spicrawl.com/llms.txt.
MCP server
Connect any MCP client to the hosted Spicrawl MCP server at https://mcp.spicrawl.com/mcp and use its 25 spicrawl_* tools to scrape, batch, manage sessions, inspect usage and search the docs.
llms.txt and Markdown docs
How an AI agent should read the Spicrawl docs: /llms.txt, /llms-full.txt, a .md version of every page, the search API, the OpenAPI spec, the Spicrawl MCP server's docs tools, and spicrawl docs in a terminal.