Factory Droid

Set up Spicrawl in Factory's Droid CLI: add the hosted MCP server with an API-key header, install the agent skill, and add AGENTS.md rules so the agent fetches web pages cheaply and handles errors correctly.

Three steps give Droid Spicrawl's tools, the knowledge to write Spicrawl code, and rules for using both well. Set your key first. Droid substitutes a header value only when the whole value is one variable, so the variable holds the full header value:

export SPICRAWL_AUTH="Bearer spicrawl_live_..."   # add to your shell profile

The spicrawl init command does not support Droid, so set it up with the steps below.

1. Add the MCP server

Spicrawl's server authenticates with an API key in the Authorization header and does not support OAuth. Droid tries OAuth discovery on remote servers by default, so add --no-oauth:

droid mcp add spicrawl https://mcp.spicrawl.com/mcp --type http --no-oauth \
  --header "Authorization: Bearer $SPICRAWL_API_KEY"

Here your shell expands $SPICRAWL_API_KEY, so Droid stores the literal key in your user config, ~/.factory/mcp.json. To keep the key out of the file, write the entry yourself and let Droid read SPICRAWL_AUTH when it connects:

~/.factory/mcp.json

{
  "mcpServers": {
    "spicrawl": {
      "type": "http",
      "url": "https://mcp.spicrawl.com/mcp",
      "headers": { "Authorization": "${SPICRAWL_AUTH}" },
      "oauth": false
    }
  }
}

Droid reads mcp.json at three levels: ~/.factory/mcp.json (you), .factory/mcp.json in an ancestor folder, and .factory/mcp.json in the project root. Factory's docs warn against putting header tokens in the project-level file, since it is committed. The ${SPICRAWL_AUTH} reference holds no secret, but keep the entry in the user file anyway so teammates are not forced to use the same variable name. Droid expands ${NAME} only in env, headers and OAuth fields, not in url, and if the variable is unset the connection fails with an error naming it.

Run /mcp in a session (or droid mcp list outside one). spicrawl should be enabled and list 25 spicrawl_* tools. See MCP server for each tool.

2. Install the skill

Droid loads a skill from any skills/<name>/SKILL.md under .factory/skills/ in the project, or ~/.factory/skills/ for every project. It also reads .agents/skills/ as a compatibility location:

mkdir -p .factory/skills/spicrawl
curl -fsSL https://app.spicrawl.com/skill.md -o .factory/skills/spicrawl/SKILL.md

Droid lists each skill's name and description and loads the full body when a task matches. See Agent skill.

3. Add rules to AGENTS.md

Droid reads AGENTS.md from the working directory up to the git root (also inside .factory/ and .agents/ at each level), plus ~/.factory/AGENTS.md for every project. Nearer files override farther ones. Paste this section into AGENTS.md at the project root:

AGENTS.md

# Web data (Spicrawl)

Use Spicrawl to read web pages: the spicrawl_* MCP tools in chat, or
POST https://api.spicrawl.com/v1/scrape with `Authorization: Bearer $SPICRAWL_API_KEY` in code.
Docs: https://docs.spicrawl.com/llms.txt (append .md to any page URL for Markdown).

- Ask for markdown: `response_format: "markdown"` (API default is html); on spicrawl_scrape, `format: "markdown"`.
- Check the site's status, not only the HTTP status: `X-Target-Status` header, or `status`
  in the JSON envelope (spicrawl_scrape with `format: "json"`). 200 is the page; 404/410
  mean it does not exist; 403/429/503 mean the site refused, so escalate.
- On an error, switch on `code`. Retry only when `retryable` is true, after
  `retry_after_seconds`. Read `diagnostics.hint` and change what it names first.
  Never retry ERR::REQUEST::*, ERR::AUTH::* or ERR::LIMIT::QUOTA_EXCEEDED.
- Escalate one step at a time and stop at the first that works:
  plain fetch (1 credit) -> `js_render: true` (3) -> add the user's own `proxy`
  if they have one. Empty or skeleton content means render; ERR::UPSTREAM::CHALLENGE
  or a 403 target status means retry once, then the user's own proxy.
- Set `max_cost` on every request to the price of the step you intend.
- Keep the cache on (default). Set `cache: false` only for prices, stock or other live data.
- Trim tokens with `main_content_only` (on by default for markdown), `include_tags`, `exclude_tags`.
- For more than 20 URLs, use a batch job (spicrawl_batch_submit / POST /v1/batch).
  Batch items return raw HTML and apply only render, proxy and block_resources settings.
- Never print or commit SPICRAWL_API_KEY. Log `X-Request-Id` for failures.

A first task to try

Start droid in the project and ask:

Use Spicrawl to read https://example.com/pricing as markdown. List each plan with its monthly price.
If the page comes back empty, retry with rendering.

The agent should call spicrawl_scrape with url and format: "markdown", and add render: true only if the first result is empty. For a coding task:

Add a fetchPage(url) function to this project that calls the Spicrawl API for markdown,
following the Web data rules in AGENTS.md. Read the key from process.env.SPICRAWL_API_KEY.

Compare the result with the reference implementation in Best practices.

Troubleshooting

Symptom	Fix
The connection fails with an error naming `SPICRAWL_AUTH`	The variable is not set in the shell that started `droid`. Export it and restart.
A 401 with `Bearer ${SPICRAWL_API_KEY}` in the header	Droid expands a variable only when it is the whole header value. Put `Bearer` and the key in one variable (`SPICRAWL_AUTH`) and use `"${SPICRAWL_AUTH}"`.
Droid opens a browser or shows an OAuth prompt for `spicrawl`	Spicrawl has no OAuth. Re-add the server with `--no-oauth`, or set `"oauth": false` in `mcp.json`.
The agent writes `js_render` on `spicrawl_scrape` and gets "unknown argument"	The tool's argument is `render`. The API field is `js_render`.
Usage tools return `ERR::AUTH::INSUFFICIENT_SCOPE`	Grant the `read` scope to the key in the dashboard.