Docs for agents

How an AI agent should read the Spicrawl docs: /llms.txt, /llms-full.txt, a .md version of every page, the search API, the OpenAPI spec, the Spicrawl MCP server's docs tools, and spicrawl docs in a terminal.

Every page of these docs is available as plain Markdown, and the site publishes an index for agents. Start an agent here:

https://docs.spicrawl.com/llms.txt

What is available

Resource	URL	What it is
Index	`https://docs.spicrawl.com/llms.txt`	One line per page: title, link to the page's `.md` version, and its one-sentence description. Also at `https://docs.spicrawl.com/.well-known/llms.txt`.
Full text	`https://docs.spicrawl.com/llms-full.txt`	Every page's content in one file, including the API reference. Also at `https://docs.spicrawl.com/.well-known/llms-full.txt`.
Markdown page	Any page URL plus `.md`, e.g. `https://docs.spicrawl.com/guides/anti-bot.md`	One page as Markdown. API reference pages include their OpenAPI operation. The docs home is `https://docs.spicrawl.com/index.md`.
Markdown by header	Any page URL with `Accept: text/markdown`	The same Markdown without changing the URL.
OpenAPI spec	`https://docs.spicrawl.com/openapi.yaml`	The API's OpenAPI document, the source of truth for every request and response field.
Search	`https://docs.spicrawl.com/api/search?query=<words>`	Full-text search as JSON: a list of results, each with `id`, `type` (`page`, `heading` or `text`), `content`, `url` and optional `breadcrumbs`.
Agent skill	`https://docs.spicrawl.com/skill.md`	The Spicrawl agent skill, the same file as `https://app.spicrawl.com/skill.md` (see Agent skill).
CLI	`spicrawl docs <topic>`	Prints a page as Markdown in a terminal.

All of these are built from the same pages you read here, on every docs deploy, and none needs an API key. Each page's HTML also links its Markdown version with <link rel="alternate" type="text/markdown">.

How an agent should read the docs

Fetch the index

curl -fsSL https://docs.spicrawl.com/llms.txt

The index is small. Each line's description says what the page answers, so the agent can choose one or two pages instead of reading everything.

Fetch only the pages it needs, as Markdown

curl -fsSL https://docs.spicrawl.com/errors.md

Pages are written to stand alone: each names the exact field names, values, limits and error codes it depends on.

Use llms-full.txt only when it fits

llms-full.txt holds every page, including the API reference. Load it when the agent has a large context window and needs broad coverage, such as writing an SDK. For a single task, the index plus two pages costs far fewer tokens.

For the request and response schema itself, the OpenAPI spec is the source of truth: fetch https://docs.spicrawl.com/openapi.yaml, or read one operation from an API reference page's .md version, for example https://docs.spicrawl.com/api-reference/introduction.md.

Search from code

curl -fsSL "https://docs.spicrawl.com/api/search?query=wait_for"

The response is a JSON array. A page result names a page; the heading and text results after it are matching sections of that page. A url is relative to the docs root: /errors#… is https://docs.spicrawl.com/errors#…. Add .md to its path (before any #anchor) to fetch the page as Markdown.

Docs tools in the Spicrawl MCP server

The Spicrawl MCP server at https://mcp.spicrawl.com/mcp includes three docs tools next to its API tools, so an agent that already has it connected needs nothing else:

Tool	What it does
`spicrawl_docs_search`	Searches these docs (`query`, optional `limit` 1-20, default 8). Results are grouped by page: title, matching sections with snippets, and the `md_url` to read next. An error code as the query (`ERR::UPSTREAM::CHALLENGE`) also returns a link to its entry on Errors.
`spicrawl_docs_read`	Reads one page as Markdown by path (`guides/anti-bot`), docs URL, or either with `#anchor`. Pages over 60,000 characters are truncated, and the text says so.
`spicrawl_docs_index`	Returns `llms.txt`: every page with its title, description and `.md` URL.

The docs tools never send your API key to the docs. Setup for each client is on MCP server.

Page actions

Each page has a Copy page button, which copies the page as Markdown, and an Open menu: View as Markdown opens the .md version, and Open in Claude, Open in ChatGPT and Open in Cursor start a chat that points the assistant at the page's Markdown.

From a terminal

The CLI prints any page as Markdown, with no API key needed:

spicrawl docs --list                      # the llms.txt index
spicrawl docs quickstart                  # one page
spicrawl docs guides/anti-bot > anti-bot.md
spicrawl docs errors --json | jq -r .markdown

A topic is the page's path without the extension. --json (or a piped stdout) wraps the page as {"topic", "url", "markdown"}. An unknown topic exits with code 2. Set SPICRAWL_DOCS_URL to a docs base URL such as https://docs.example.test (a self-hosted API's docs: http://<host>:8080/docs) to read from another docs host.

Tell your agent where the docs are

Add one line to your agent's instructions file (CLAUDE.md, AGENTS.md, or a Cursor rule):

Spicrawl docs: fetch https://docs.spicrawl.com/llms.txt, then the .md version of the page you need. Do not guess field names; the API rejects unknown fields with ERR::REQUEST::INVALID_PARAMETER.

Docs for agents

On this page