Get past anti-bot challenges

Use this when a scrape fails with ERR::UPSTREAM::CHALLENGE, or when the content you get back is a "Just a moment…" or "Access denied" page. Sites behind Cloudflare, DataDome, Akamai or PerimeterX score each request on its IP, its TLS fingerprint and its browser behaviour, and serve an interstitial instead of the page when the score is low. A challenge is never billed: you pay 0 credits until a request returns the real page.

The escalation ladder

Climb one rung at a time. Each rung fixes a different signal, and the cheapest one that works is the one to keep.

Two more rungs are on the way: stealth mode Coming soon, which renders in the hardened Camoufox browser, and Spicrawl's managed residential exits through premium_proxy Coming soon. Through your own proxy or a premium_proxy exit, a challenged request can also climb to the stealth browser by itself. If you do not want to choose between rungs 1 and 2, mode=auto walks fetch → obscura for you and bills only the rung that worked; see JavaScript rendering.

curl https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products/42", "js_render": true, "proxy": "http://user:pass@proxy.example.net:8000", "max_cost": 3}'

What a challenge looks like

HTTP 502, application/problem+json, 0 credits:

{
  "type": "https://docs.spicrawl.com/errors#UPSTREAM_CHALLENGE",
  "title": "Target served a bot challenge",
  "status": 502,
  "code": "ERR::UPSTREAM::CHALLENGE",
  "detail": "Cloudflare served a bot challenge instead of the page, so no content could be fetched (...). Nothing was charged.",
  "retryable": true,
  "target_status": 403,
  "request_id": "01JAX3K6Q8V1T7M2C9D4E5F6G7",
  "diagnostics": {
    "failed_at": "fetch",
    "engine": "fetch",
    "proxy": { "source": "direct" },
    "hint": "Every engine tier was served a bot challenge rather than the page. Nothing was charged. The verdict is per exit and per moment, so a retry often passes."
  }
}

A request fails with ERR::UPSTREAM::CHALLENGE when the response carries a marker that names the vendor (for example Cloudflare's cf-mitigated header or DataDome's captcha-delivery.com script). Under mode=auto, any detected block escalates to the next rung, and the error is returned only when every rung was challenged. On your own proxy or a premium_proxy exit, only a challenge that names its vendor escalates; see Automatic escalation.

A bare 403 or 429 is the target's answer

A 403, 429 or 503 with no vendor marker is not reported as a challenge: it looks the same as a login wall, a geo-block or a down origin. You get HTTP 200 with the site's own body, X-Target-Status: 403 (or 429, 503) and X-Credits-Charged: 0. Treat it as the site's decision. A 429 from the target means you are sending that site too much traffic; slow down rather than climbing the ladder.

Automatic escalation Coming soon

When the page comes back as a challenge that names its vendor (Cloudflare, DataDome, PerimeterX or Akamai), Spicrawl tries to get past it before returning an error:

1. A different exit. A plain fetch through a managed pool exit is retried once on another exit, free. It is skipped for sticky_key and session_id, which fix the exit, for your own proxy, and for methods other than GET, HEAD and OPTIONS.
2. The stealth browser. A request that goes out through your own proxy or a premium_proxy exit, pins no engine and does not use mode=auto moves to Camoufox if the page is still a challenge. Camoufox waits out the interstitial or clicks its checkbox before capturing the page.

You pay for the rung that returned the page: 25 credits if Camoufox did, the fetch or obscura price if an earlier rung did, and 0 if every rung was challenged (ERR::UPSTREAM::CHALLENGE).

The request does not move to Camoufox on a transport error, on a refusal with no vendor marker (a bare 403), or on a non-billable target status such as 404: those come back as they would without it. The move to Camoufox is off when:

• max_cost is below 25;
• method is not GET;
• the request sets headless=false, response_format=pdf, a screenshot or actions;
• your proxy is not an http:// URL (the browser can only use an http:// proxy);
• the deployment does not run Camoufox.

Before the request runs, the 25 credits are reserved against your monthly allowance. If the allowance covers the first rung but not 25, the request runs without the move to Camoufox.

The response says when it escalated: X-Engine names the engine that served, each abandoned rung adds an X-Warning: ESCALATED with its reason, and diagnostics.attempts (in an error body, or the JSON envelope after a successful climb) lists every attempt, including a retry on a different exit. Batch items never escalate this way, because a batch prices each item when it is accepted.

Why datacenter IPs get challenged

Anti-bot vendors keep reputation lists of datacenter IP ranges and challenge them on sight, however well the browser behaves. That is why rendering alone often still fails, and why rung 3 usually matters most. Supplying your own residential proxy through proxy works at no proxy surcharge; see Proxies and geo.

Retry before you climb

ERR::UPSTREAM::CHALLENGE is retryable: true. The verdict is per exit and per moment, so one or two retries on the same rung are free when they fail and often pass. Use --retry N on the CLI.

Reading blocks in your request logs

A request that failed with ERR::UPSTREAM::CHALLENGE is logged with status: "blocked", credits_micro: 0 and a blocked object:

{
  "id": "01JAX3K6Q8V1T7M2C9D4E5F6G7",
  "status": "blocked",
  "error_code": "ERR::UPSTREAM::CHALLENGE",
  "http_status": 403,
  "engine": "fetch",
  "proxy": { "source": "direct" },
  "blocked": { "vendor": "cloudflare", "signal": "cf-mitigated response header", "rule": "decisive-signal" },
  "credits_micro": 0
}

vendor is one of cloudflare, datadome, akamai, perimeterx or generic. signal names the observable marker, so you can confirm it in the response yourself. rule is the detection path: decisive-signal (a vendor-specific interstitial marker), phrase-and-status (challenge wording on a 403/429/503), phrase-and-vendor (challenge wording plus a vendor marker in a page under 8 KiB), or empty-refusal (a 403/429/503 with almost no visible text). List blocks with GET /v1/requests?status=blocked or spicrawl logs --status blocked, and open one with spicrawl logs get <request-id>.

Failure modes

Cost

Challenges, timeouts and non-billable target statuses cost 0. A success is billed at the engine that produced it: 1 for fetch, 3 for obscura, 8 for chromium, and 25 when an automatic escalation reached Camoufox Coming soon. Your own proxy adds nothing. impersonate is on by default and free. See Credits.

Related

• Proxies and geo
• JavaScript rendering
• Sessions and logins to keep cookies once you are through
• Errors