AI app builders

Two separate things can use Spicrawl in a prompt-to-app builder:

• The builder's agent reads web pages while it works (docs, competitor sites, content to copy). You connect the hosted MCP server, https://mcp.spicrawl.com/mcp, to the builder. It exposes 25 spicrawl_* tools. See MCP server.
• The app it builds fetches pages at runtime. It calls POST https://api.spicrawl.com/v1/scrape from a server route. See Calling Spicrawl from the app you build.

The MCP server authenticates with Authorization: Bearer <your Spicrawl API key>. It does not support OAuth, so a builder that only offers OAuth for custom servers cannot connect. Unauthenticated requests get 401 Unauthorized: send your Spicrawl API key.

Connect the MCP server

{
  "displayName": "Spicrawl",
  "baseUrl": "https://mcp.spicrawl.com/mcp"
}

Check the connection by asking the builder: "Use Spicrawl to read https://example.com/pricing as markdown and list each plan with its monthly price." The agent should call spicrawl_scrape. If it does not have the tools, or the connector reports 401, the key is missing or wrong.

Calling Spicrawl from the app you build

The app calls the scrape API from a server route, with the key read from the platform's secrets or environment variables as SPICRAWL_API_KEY. The browser calls your route, never Spicrawl.

export async function POST(req: Request) {
  const { url } = await req.json();

  const res = await fetch("https://api.spicrawl.com/v1/scrape", {
    method: "POST",
    headers: {
      Authorization: `Bearer ${process.env.SPICRAWL_API_KEY}`,
      "Content-Type": "application/json",
    },
    // max_cost: refuse anything above a plain fetch (1 credit) before it runs.
    body: JSON.stringify({ url, response_format: "markdown", max_cost: 1 }),
  });

  if (!res.ok) {
    // Spicrawl error bodies carry a `code`; failed requests cost 0 credits.
    return Response.json(await res.json(), { status: res.status });
  }

  return Response.json({
    targetStatus: Number(res.headers.get("X-Target-Status")),
    markdown: await res.text(),
  });
}

Details the generated code must get right:

• response_format defaults to html. Send "markdown" for text you can pass to a model or render.
• A 200 from Spicrawl means Spicrawl fetched the page. The site's own status is the X-Target-Status header, so check it before trusting the content.
• max_cost: 1 caps each request at a plain fetch. A page that needs JavaScript rendering (3 credits) is refused with ERR::LIMIT::MAX_COST_EXCEEDED at 0 credits; raise the cap and add js_render: true only for those pages.
• The route spends your credits on any URL it is sent. Put it behind your app's sign-in, or rate-limit it, before you publish the app.

Prompt to give the builder:

Add a server route that fetches a URL through Spicrawl and returns it as markdown.
Call POST https://api.spicrawl.com/v1/scrape with the JSON body
{"url": <the URL>, "response_format": "markdown", "max_cost": 1} and the header
Authorization: Bearer followed by process.env.SPICRAWL_API_KEY.
Read the key only on the server from the environment variable SPICRAWL_API_KEY, never
in client components, never in the browser bundle, and never hard-coded.
Return the X-Target-Status response header alongside the markdown.
If Spicrawl responds with an error, pass its JSON error body and status back to the caller.
Only signed-in users may call the route.
Add a form that posts a URL to this route and shows the markdown.

Then add SPICRAWL_API_KEY in the builder's secrets or environment settings (see each tab above) before you run the app. For Next.js, do not prefix the name with NEXT_PUBLIC_: that would put the key in the browser bundle.

Next steps

• Quickstart: the full scrape request and response.
• Errors: every code and what to do about it.
• Best practices: retries, escalation and cost limits for agent-written code.
• MCP server: the 25 spicrawl_* tools.