Devin

Set up Spicrawl in Devin: add the hosted MCP server as a custom MCP with an Auth Header, store the key in Devin Secrets, and give Devin a skill so it fetches web pages cheaply and handles errors correctly.

Devin (Cognition's cloud agent at app.devin.ai) runs in its own cloud environment, so nothing on your machine is visible to it. Three steps give it Spicrawl's tools, the knowledge to write Spicrawl code, and rules for using both well. You need an API key from app.spicrawl.com (spicrawl_live_...).

1. Add the MCP server

Only organization admins with the Manage MCP Servers permission can add a custom MCP. Members without it can use Suggest MCP Integration to ask an admin.

Open the MCP settings

Go to Customize → MCPs, select Add MCP, then Add custom MCP.

Enter the server

Choose the HTTP transport (Streamable HTTP) and set the values below. Spicrawl's server does not support OAuth, so pick Auth Header, not OAuth.

Field	Value
Server URL	`https://mcp.spicrawl.com/mcp`
Authentication	Auth Header
Header key	`Authorization` (the default)
Header value	`Bearer spicrawl_live_...` (your key)

Save and check

The server should list 25 spicrawl_* tools. See MCP server for each tool.

Devin's Browse marketplace (in Customize → MCPs) lists ready-made MCP servers. Spicrawl is not listed there, so add it as a custom MCP.

2. Store the key as a secret

Code Devin runs (for example a script that calls https://api.spicrawl.com) reads the key from an environment variable, not from the MCP header. Add it under Settings → Resources → Secrets (app.devin.ai/secrets) with the name SPICRAWL_API_KEY. Devin binds secrets as environment variables when it runs shell commands. An organization secret is usable by every member (only admins can view or edit it); a personal secret applies to your own sessions only.

Devin's documentation does not say whether a secret can fill the MCP header value, so paste the key into the header field in step 1.

3. Install the skill

Devin reads skills from .agents/skills/<name>/SKILL.md in the repository (its recommended location). It also scans .devin/skills/ and .github/skills/. Commit the Spicrawl skill:

mkdir -p .agents/skills/spicrawl
curl -fsSL https://app.spicrawl.com/skill.md -o .agents/skills/spicrawl/SKILL.md

Devin invokes a skill when it matches the task, or when you mention it as @skills:spicrawl. See Agent skill.

Devin's older Knowledge feature is being migrated to Skills. Playbooks, which live in the Devin web app and apply across the organization, are the alternative when you do not want to commit files to a repository. Step 4 uses one.

4. Add instructions

Save this as a Devin playbook (or append it to the skill's SKILL.md). The rules are the same as in Cursor and Codex.

Playbook or instructions

# Web data (Spicrawl)

Use Spicrawl to read web pages: the spicrawl_* MCP tools in chat, or
POST https://api.spicrawl.com/v1/scrape with `Authorization: Bearer $SPICRAWL_API_KEY` in code.
Docs: https://docs.spicrawl.com/llms.txt (append .md to any page URL for Markdown).

- Ask for markdown: `response_format: "markdown"` (API default is html); on spicrawl_scrape, `format: "markdown"`.
- Check the site's status, not only the HTTP status: `X-Target-Status` header, or `status`
  in the JSON envelope (spicrawl_scrape with `format: "json"`). 200 is the page; 404/410
  mean it does not exist; 403/429/503 mean the site refused, so escalate.
- On an error, switch on `code`. Retry only when `retryable` is true, after
  `retry_after_seconds`. Read `diagnostics.hint` and change what it names first.
  Never retry ERR::REQUEST::*, ERR::AUTH::* or ERR::LIMIT::QUOTA_EXCEEDED.
- Escalate one step at a time and stop at the first that works:
  plain fetch (1 credit) -> `js_render: true` (3) -> add the user's own `proxy`
  if they have one. Empty or skeleton content means render; ERR::UPSTREAM::CHALLENGE
  or a 403 target status means retry once, then the user's own proxy.
- Set `max_cost` on every request to the price of the step you intend.
- Keep the cache on (default). Set `cache: false` only for prices, stock or other live data.
- Trim tokens with `main_content_only` (on by default for markdown), `include_tags`, `exclude_tags`.
- For more than 20 URLs, use a batch job (spicrawl_batch_submit / POST /v1/batch).
  Batch items return raw HTML and apply only render, proxy and block_resources settings.
- Never print or commit SPICRAWL_API_KEY. Log `X-Request-Id` for failures.

A first task to try

Start a Devin session and write:

Use Spicrawl to read https://example.com/pricing as markdown. List each plan with its monthly price.
If the page comes back empty, retry with rendering.

Devin should call spicrawl_scrape with url and format: "markdown", and add render: true only if the first result is empty. For a coding task:

Add a fetchPage(url) function to this repository that calls the Spicrawl API for markdown,
following the Spicrawl skill's error-handling steps. Read the key from process.env.SPICRAWL_API_KEY.

Compare the result with the reference implementation in Best practices.

Troubleshooting

Symptom	Fix
Add custom MCP is missing	You lack the Manage MCP Servers permission. Ask an admin, or use Suggest MCP Integration.
The server returns 401	The header value must be `Bearer` followed by the key, with header key `Authorization`. Check for a trailing space or a revoked key.
Devin's own code gets HTTP 401 from the API	`SPICRAWL_API_KEY` is not set as a secret. Add it under Settings → Resources → Secrets.
The agent writes `js_render` on `spicrawl_scrape` and gets "unknown argument"	The tool's argument is `render`. The API field is `js_render`.
Usage tools return `ERR::AUTH::INSUFFICIENT_SCOPE`	Grant the `read` scope to the key in the dashboard.

Devin

On this page