Spicrawl

Spicrawl turns any URL into data your code or AI agent can use: clean markdown, HTML, text, a PDF, or JSON extracted with selectors (extraction with a model Coming soon). You send one request shape to https://api.spicrawl.com/v1/scrape, say what you need with flags (js_render, proxy, extract), and Spicrawl picks the rendering path and retries for you. You pay credits only for successful requests; failures cost 0, and a cache hit is billed at the same price as the fetch that stored it.

curl -sS "https://api.spicrawl.com/v1/scrape" \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products/42", "response_format": "markdown"}'

The response body is the page as markdown. The headers say what the site answered (X-Target-Status), which rendering tier ran (X-Engine) and what it cost (X-Credits-Charged).

Cloudflare-protected pages are now supported when you bring your own proxy (residential works best); automatic handling on managed premium exits is coming soon Coming soon. See Anti-bot.

For AI agents

Agents can use Spicrawl, and read these docs, without a human in the loop:

Rules an agent should hold on to:

• Two statuses. The HTTP status is Spicrawl's. The site's status is X-Target-Status. A site that refused you is still HTTP 200, at 0 credits.
• Switch on code, retry on retryable. Errors are application/problem+json. Honour Retry-After, and read diagnostics.hint: it names the parameter to change. See Errors.
• Failures cost 0. X-Credits-Charged is what was billed.
• Not idempotent. Retrying a POST /v1/scrape that succeeded performs and bills a second scrape.
• Unknown fields are rejected. A misspelled parameter is a 400, never ignored.