Get past anti-bot challenges

Use this when a scrape fails with ERR::UPSTREAM::CHALLENGE, or when the content you get back is a "Just a moment…" or "Access denied" page. Sites behind Cloudflare, DataDome, Akamai, PerimeterX, Kasada or Imperva score each request on its IP, its TLS fingerprint and its browser behaviour, and serve an interstitial instead of the page when the score is low. A challenge is never billed: you pay 0 credits until a request returns the real page.

Cloudflare-protected pages

Cloudflare-protected pages work when you bring your own proxy (a residential proxy works best). When a page answers with a Cloudflare challenge, or a similar managed challenge from another vendor, Spicrawl moves the request to a stronger rendering path on the same proxy exit and returns the real page. You do not pick a browser or set a rendering flag for this: a plain request and a js_render request both end up there. A challenged page typically takes about 20 seconds end to end, so allow for it in your client's timeout. Whether a given page clears still depends on the site and the exit, so treat it as likely rather than guaranteed. If a request uses your own proxy and hits a blocking challenge that Spicrawl solves, your proxy's address and credentials are shared with the third-party solving service for that solve.

Pass your proxy in proxy as an http:// URL. Automatic handling on managed premium exits (premium_proxy) Coming soon is coming soon. Without a proxy of your own, a challenged page can return ERR::UPSTREAM::CHALLENGE instead.

For a step-by-step walkthrough, see How to scrape Cloudflare-protected pages with Spicrawl.

curl https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products/42", "proxy": "http://user:pass@proxy.example.net:8000", "response_format": "json"}'

{
  "url": "https://example.com/products/42",
  "final_url": "https://example.com/products/42",
  "status": 200,
  "content": "<!doctype html>…",
  "credits": 8,
  "proxy_source": "custom"
}

You pay for the rendering tier that returned the page, with no proxy surcharge: 1 credit if the plain fetch passed, 8 if a challenge moved the request to full-browser rendering and that tier served the page, and 0 if every rung was challenged. A response that climbed carries X-Warning: ESCALATED. Add proxy_country with the exit's country so the render's clock and language match the IP. Set max_cost to cap the spend; a rung dearer than it is left off the climb. The steps are under Automatic escalation.

If you get ERR::UPSTREAM::CHALLENGE

The error is retryable and costs 0 credits.

1. Retry with your own residential proxy. Pass it in proxy as an http:// URL, as in the example above. A datacenter exit is challenged on sight by most vendors, and a residential one passes far more often.
2. Retry once or twice. The verdict is per exit and per moment, so a retry often passes.
3. Check what was served. If your own residential proxy is still challenged, the site is showing a challenge that automation does not always clear. Open the request in your logs to see which vendor and signal it was; see Reading blocks in your request logs.

Spicrawl's managed residential exits (premium_proxy) Coming soon will be another way to do step 1.

The escalation ladder

Climb one rung at a time. Each rung fixes a different signal, and the cheapest one that works is the one to keep.

JavaScript rendering runs the page's scripts, but some managed interstitials only clear in full-browser rendering, which runs with a real display. You do not choose between them: the engine and headless fields are deprecated and ignored (the response carries X-Warning: ENGINE_IGNORED), and a challenged request moves up by itself on rung 3 (and on rung 4 once it is available); see Automatic escalation. Stealth rendering Coming soon, a hardened tier above full-browser rendering, is also reached this way. If you do not want to choose between rungs 1 and 2, mode=auto walks from a plain fetch to JavaScript rendering to full-browser rendering for you, moving up only when the previous rung was met by a challenge, and bills only the rung that worked; see JavaScript rendering.

curl https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products/42", "js_render": true, "proxy": "http://user:pass@proxy.example.net:8000", "max_cost": 3}'

What a challenge looks like

HTTP 502, application/problem+json, 0 credits:

{
  "type": "https://docs.spicrawl.com/errors#UPSTREAM_CHALLENGE",
  "title": "Target served a bot challenge",
  "status": 502,
  "code": "ERR::UPSTREAM::CHALLENGE",
  "detail": "Cloudflare served a bot challenge instead of the page, so no content could be fetched (...). Nothing was charged.",
  "retryable": true,
  "target_status": 403,
  "request_id": "01JAX3K6Q8V1T7M2C9D4E5F6G7",
  "diagnostics": {
    "failed_at": "fetch",
    "proxy": { "source": "direct" },
    "hint": "Every engine tier was served a bot challenge rather than the page. Nothing was charged. The verdict is per exit and per moment, so a retry often passes."
  }
}

A request fails with ERR::UPSTREAM::CHALLENGE when the response carries a marker that names the vendor (for example Cloudflare's cf-mitigated header or DataDome's captcha-delivery.com script). Under mode=auto, any detected block escalates to the next rung, and the error is returned only when every rung was challenged. On your own proxy (or a premium_proxy Coming soon exit), only a challenge that names its vendor escalates; see Automatic escalation. What to do next is under If you get ERR::UPSTREAM::CHALLENGE.

A bare 403 or 429 is the target's answer

A plain fetch that returns a near-empty HTML page (under 100 visible characters) that runs a script or meta-refreshes itself is reported the same way, at 0 credits. A short page with no script is returned as the page.

A 403, 429 or 503 with no vendor marker is not reported as a challenge: it looks the same as a login wall, a geo-block or a down origin. You get HTTP 200 with the site's own body, X-Target-Status: 403 (or 429, 503) and X-Credits-Charged: 0. Treat it as the site's decision. The same goes for a CDN's own error or geo notice, such as CloudFront's "ERROR: The request could not be satisfied" or a "Not available in your region" page: even when it arrives with a 200, it is returned with X-Target-Status: 403 and no charge. A 429 from the target means you are sending that site too much traffic; slow down rather than climbing the ladder.

Automatic escalation

When the page comes back as a challenge that names its vendor (Cloudflare, DataDome, PerimeterX, Akamai, Kasada or Imperva), Spicrawl tries to get past it before returning an error:

1. A different exit. A plain fetch through a managed premium exit Coming soon is retried once on another exit, free. It is skipped for sticky_key and session_id, which fix the exit, for your own proxy, and for methods other than GET, HEAD and OPTIONS.
2. Stronger rendering. A request that goes out through your own proxy (or a premium_proxy Coming soon exit), has no session_id, does not use mode=auto, and starts on a plain fetch or JavaScript rendering, moves to full-browser rendering and then to stealth rendering Coming soon, one rung at a time, while the page is still a challenge. Full-browser rendering runs with a real display, which clears managed interstitials that JavaScript rendering cannot. It waits out the interstitial before capturing the page, which typically takes about 15 to 20 seconds on your own proxy. This is how Cloudflare-protected pages are handled.

If full-browser rendering is itself met by a challenge that names its vendor, the request ends there: it returns ERR::UPSTREAM::CHALLENGE at 0 credits and does not try stealth rendering Coming soon. A vendor that has refused a browser with a real display has not been beaten by a second browser, and the attempt would only spend what is left of the request's time budget. The skipped step is listed in X-Warning: ESCALATED (skipped: Cloudflare already refused a full browser) and in diagnostics.attempts. A full-browser rung that fails without naming a vendor (a bare 403, a timeout) still climbs on to stealth rendering.

The steps share one time budget per request. The steps before full-browser rendering are cut short, or skipped with the reason in X-Warning: ESCALATED, so that full-browser rendering keeps the time it needs to wait out an interstitial.

You pay for the rung that returned the page: 8 credits if full-browser rendering did, 25 if stealth rendering did, the first rung's price (1 or 3) if it did, and 0 if every rung was challenged (ERR::UPSTREAM::CHALLENGE). A premium_proxy Coming soon exit prices these tiers at 10, 25 and 32.

On a full-browser render, Spicrawl can also solve a blocking challenge from Cloudflare, where the platform offers solving. DataDome pages are not solved. This applies to a request with your own proxy and to one without it. If a request uses your own proxy and hits a blocking challenge that Spicrawl solves, your proxy's address and credentials are shared with the third-party solving service for that solve. Without your own proxy, a blocked request may be completed through a residential exit of ours at no change in price (only the 5-credit solve surcharge applies, when a solve succeeds). A solve adds 5 credits to the rung that served the page, once per request, and only when it succeeded. CAPTCHA widgets are solved only when the request sets solve_captcha: true, as described in Cloudflare-protected pages. A solved token is put into the page, but the site decides whether to accept it.

An incomplete page climbs the same way: on a multi-rung plan (mode=auto, or a premium_proxy Coming soon/own-proxy challenge ladder), a 2xx HTML document whose URL claims it is a search page (a ?q=-style query or a /search path) but carries fewer than 10 item links and fewer than 4 prices, or an item page (a long-id/slug path) with no price and under 1,500 characters of visible text, abandons the rung and the ladder climbs, billed like a challenge. A pinned single-rung request returns the document as before; diagnostics report vendor incomplete, rule incomplete-page.

The request does not climb on a transport error, on a refusal with no vendor marker (a bare 403), or on a non-billable target status such as 404: those come back as they would without it. A rung is left off when:

• max_cost is below that rung's price;
• method is not GET;
• the request sets actions (a screenshot or response_format=pdf already starts on full-browser rendering, so there is no climb to make);
• your proxy is not an http:// URL (rendering can only use an http:// proxy);
• the deployment does not run that rung.

Before the request runs, the dearest rung's credits (25 on your own proxy) are reserved against your monthly allowance. If the allowance covers the first rung but not that, the request runs without the climb.

The response says when it escalated: each abandoned rung adds an X-Warning: ESCALATED naming the tier, its price and the reason (ESCALATED: fetch (1 credit): blocked by challenge page (Cloudflare …)), X-Engine names the tier that served, X-Credits-Charged carries the price of the rung that served, and diagnostics.attempts (in an error body, or the JSON envelope after a successful climb) lists every attempt, including a retry on a different exit. Batch items never escalate this way, because a batch prices each item when it is accepted.

Why datacenter IPs get challenged

Anti-bot vendors keep reputation lists of datacenter IP ranges and challenge them on sight, however well the browser behaves. That is why rendering alone often still fails, and why rung 3 usually matters most. Supply your own residential proxy through proxy at no proxy surcharge; Spicrawl's managed residential exits (premium_proxy) Coming soon will be another option. See Proxies and geo.

Retry before you climb

ERR::UPSTREAM::CHALLENGE is retryable: true. The verdict is per exit and per moment, so one or two retries on the same rung are free when they fail and often pass. Use --retry N on the CLI.

Reading blocks in your request logs

A request that failed with ERR::UPSTREAM::CHALLENGE is logged with status: "blocked", credits_micro: 0 and a blocked object:

{
  "id": "01JAX3K6Q8V1T7M2C9D4E5F6G7",
  "status": "blocked",
  "error_code": "ERR::UPSTREAM::CHALLENGE",
  "http_status": 403,
  "proxy": { "source": "direct" },
  "blocked": { "vendor": "cloudflare", "signal": "cf-mitigated response header", "rule": "decisive-signal" },
  "credits_micro": 0
}

vendor is one of cloudflare, datadome, akamai, perimeterx, kasada, imperva, reddit, empty-shell or generic. reddit is Reddit's block page and its JS challenge. empty-shell is a page that is only script, with a title at most and nothing to read. signal names the observable marker, so you can confirm it in the response yourself. rule is the detection path: decisive-signal (a vendor-specific interstitial marker), phrase-and-status (challenge wording on a 403/429/503), phrase-and-vendor (challenge wording plus a vendor marker in a page under 8 KiB), vendor-page (a vendor's own interstitial page, such as Akamai's "Access Denied" reference page or Imperva's "Pardon Our Interruption", which can arrive as a 200), empty-shell (a 200 page that is nearly all script), empty-refusal (a 403/429/503 with almost no visible text), or thin-content (a 200 script-driven page under 100 visible characters, reported with vendor empty-shell). List blocks with GET /v1/requests?status=blocked or spicrawl logs --status blocked, and open one with spicrawl logs get <request-id>.

Failure modes

Cost

Challenges, timeouts and non-billable target statuses cost 0. A success is billed at the rendering tier that produced it: 1 for a plain fetch, 3 for JavaScript rendering, 8 for full-browser rendering, and 25 when an automatic escalation reached stealth rendering Coming soon. On a premium_proxy Coming soon exit these are 10, 25 and 32, with stealth rendering staying at 25. A solve adds 5 on top, once per request, only when it succeeded. Your own proxy adds nothing. impersonate is on by default and free. See Credits.

Related

• Proxies and geo
• JavaScript rendering
• Sessions and logins to keep cookies once you are through
• Errors