# Docs for agents

> How an AI agent should read the Spicrawl docs: /llms.txt, /llms-full.txt, a .md version of every page, the search API, the OpenAPI spec, the Spicrawl MCP server's docs tools, and spicrawl docs in a terminal.

Source: https://docs.spicrawl.com/agents/llms-txt

Every page of these docs is available as plain Markdown, and the site publishes an index for agents. Start an agent here:

```text
https://docs.spicrawl.com/llms.txt
```

## What is available

| Resource           | URL                                                                          | What it is                                                                                                                                              |
| ------------------ | ---------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Index              | `https://docs.spicrawl.com/llms.txt`                                         | One line per page: title, link to the page's `.md` version, and its one-sentence description. Also at `https://docs.spicrawl.com/.well-known/llms.txt`. |
| Full text          | `https://docs.spicrawl.com/llms-full.txt`                                    | Every page's content in one file, including the API reference. Also at `https://docs.spicrawl.com/.well-known/llms-full.txt`.                           |
| Markdown page      | Any page URL plus `.md`, e.g. `https://docs.spicrawl.com/guides/anti-bot.md` | One page as Markdown. API reference pages include their OpenAPI operation. The docs home is `https://docs.spicrawl.com/index.md`.                       |
| Markdown by header | Any page URL with `Accept: text/markdown`                                    | The same Markdown without changing the URL.                                                                                                             |
| OpenAPI spec       | `https://docs.spicrawl.com/openapi.yaml`                                     | The API's OpenAPI document, the source of truth for every request and response field.                                                                   |
| Search             | `https://docs.spicrawl.com/api/search?query=<words>`                         | Full-text search as JSON: a list of results, each with `id`, `type` (`page`, `heading` or `text`), `content`, `url` and optional `breadcrumbs`.         |
| Agent skill        | `https://docs.spicrawl.com/skill.md`                                         | The Spicrawl agent skill, the same file as `https://app.spicrawl.com/skill.md` (see [Agent skill](https://docs.spicrawl.com/agents/skill.md)).                                      |
| CLI                | `spicrawl docs <topic>`                                                      | Prints a page as Markdown in a terminal.                                                                                                                |

All of these are built from the same pages you read here, on every docs deploy, and none needs an API key. Each page's HTML also links its Markdown version with `<link rel="alternate" type="text/markdown">`.

## How an agent should read the docs

**Step 1: Fetch the index**

```bash
curl -fsSL https://docs.spicrawl.com/llms.txt
```

The index is small. Each line's description says what the page answers, so the agent can choose one or two pages instead of reading everything.

**Step 2: Fetch only the pages it needs, as Markdown**

```bash
curl -fsSL https://docs.spicrawl.com/errors.md
```

Pages are written to stand alone: each names the exact field names, values, limits and error codes it depends on.

**Step 3: Use llms-full.txt only when it fits**

`llms-full.txt` holds every page, including the API reference. Load it when the agent has a large context window and needs broad coverage, such as writing an SDK. For a single task, the index plus two pages costs far fewer tokens.

For the request and response schema itself, the OpenAPI spec is the source of truth: fetch `https://docs.spicrawl.com/openapi.yaml`, or read one operation from an API reference page's `.md` version, for example `https://docs.spicrawl.com/api-reference/introduction.md`.

## Search from code

```bash
curl -fsSL "https://docs.spicrawl.com/api/search?query=wait_for"
```

The response is a JSON array. A `page` result names a page; the `heading` and `text` results after it are matching sections of that page. A `url` is relative to the docs root: `/errors#…` is `https://docs.spicrawl.com/errors#…`. Add `.md` to its path (before any `#anchor`) to fetch the page as Markdown.

## Docs tools in the Spicrawl MCP server

The [Spicrawl MCP server](https://docs.spicrawl.com/agents/mcp.md) at `https://mcp.spicrawl.com/mcp` includes three docs tools next to its API tools, so an agent that already has it connected needs nothing else:

| Tool                   | What it does                                                                                                                                                                                                                                                                        |
| ---------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `spicrawl_docs_search` | Searches these docs (`query`, optional `limit` 1-20, default 8). Results are grouped by page: title, matching sections with snippets, and the `md_url` to read next. An error code as the query (`ERR::UPSTREAM::CHALLENGE`) also returns a link to its entry on [Errors](https://docs.spicrawl.com/errors.md). |
| `spicrawl_docs_read`   | Reads one page as Markdown by path (`guides/anti-bot`), docs URL, or either with `#anchor`. Pages over 60,000 characters are truncated, and the text says so.                                                                                                                       |
| `spicrawl_docs_index`  | Returns `llms.txt`: every page with its title, description and `.md` URL.                                                                                                                                                                                                           |

The docs tools never send your API key to the docs. Setup for each client is on [MCP server](https://docs.spicrawl.com/agents/mcp.md).

## Page actions

Each page has a **Copy page** button, which copies the page as Markdown, and an **Open** menu: **View as Markdown** opens the `.md` version, and **Open in Claude**, **Open in ChatGPT** and **Open in Cursor** start a chat that points the assistant at the page's Markdown.

## From a terminal

The CLI prints any page as Markdown, with no API key needed:

```bash
spicrawl docs --list                      # the llms.txt index
spicrawl docs quickstart                  # one page
spicrawl docs guides/anti-bot > anti-bot.md
spicrawl docs errors --json | jq -r .markdown
```

A topic is the page's path without the extension. `--json` (or a piped stdout) wraps the page as `{"topic", "url", "markdown"}`. An unknown topic exits with code `2`. Set `SPICRAWL_DOCS_URL` to a docs base URL such as `https://docs.example.test` (a self-hosted API's docs: `http://<host>:8080/docs`) to read from another docs host.

## Tell your agent where the docs are

Add one line to your agent's instructions file (`CLAUDE.md`, `AGENTS.md`, or a Cursor rule):

```markdown
Spicrawl docs: fetch https://docs.spicrawl.com/llms.txt, then the .md version of the page you need. Do not guess field names; the API rejects unknown fields with ERR::REQUEST::INVALID_PARAMETER.
```