# Get past anti-bot challenges > What ERR::UPSTREAM::CHALLENGE means, the escalation ladder from a browser-fingerprinted fetch to rendering to your own proxy with the credit cost of each rung, the automatic climb to the stealth browser, and how to read blocks in your request logs. Source: https://docs.spicrawl.com/guides/anti-bot Use this when a scrape fails with `ERR::UPSTREAM::CHALLENGE`, or when the content you get back is a "Just a moment…" or "Access denied" page. Sites behind Cloudflare, DataDome, Akamai or PerimeterX score each request on its IP, its TLS fingerprint and its browser behaviour, and serve an interstitial instead of the page when the score is low. A challenge is never billed: you pay 0 credits until a request returns the real page. ## The escalation ladder Climb one rung at a time. Each rung fixes a different signal, and the cheapest one that works is the one to keep. | Rung | Add | Fixes | Credits per success | | ---- | --------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------- | | 1 | nothing: `impersonate` is on by default | Non-browser TLS/HTTP2 fingerprint. Every fetch presents a current Chrome fingerprint with matching headers. Send `impersonate: false` only to turn it off. | 1 (no surcharge) | | 2 | `js_render: true` | JavaScript checks the page runs before showing content. | 3 | | 3 | `proxy` (your own) | IP reputation. Route through a proxy you control, for example a residential one. | engine price, no proxy surcharge | Two more rungs are on the way: stealth mode (coming soon), which renders in the hardened Camoufox browser, and Spicrawl's [managed residential exits](https://docs.spicrawl.com/guides/proxies-and-geo.md#managed-proxy-pool-coming-soon) through `premium_proxy` (coming soon). Through your own `proxy` or a `premium_proxy` exit, a challenged request can also [climb to the stealth browser by itself](#automatic-escalation-coming-soon). If you do not want to choose between rungs 1 and 2, `mode=auto` walks fetch → obscura for you and bills only the rung that worked; see [JavaScript rendering](https://docs.spicrawl.com/guides/javascript-rendering.md). ```bash title="curl" curl https://api.spicrawl.com/v1/scrape \ -H "Authorization: Bearer $SPICRAWL_API_KEY" \ -H "Content-Type: application/json" \ -d '{"url": "https://example.com/products/42", "js_render": true, "proxy": "http://user:pass@proxy.example.net:8000", "max_cost": 3}' ``` ```python title="Python" import os, requests r = requests.post( "https://api.spicrawl.com/v1/scrape", headers={"Authorization": f"Bearer {os.environ['SPICRAWL_API_KEY']}"}, json={"url": "https://example.com/products/42", "js_render": True, "proxy": os.environ["MY_PROXY_URL"], "max_cost": 3}, timeout=180, ) if r.status_code != 200: problem = r.json() print(problem["code"], problem["retryable"], problem.get("detail")) else: print(r.headers["X-Engine"], r.headers["X-Target-Status"], len(r.text)) ``` ```typescript title="TypeScript" const r = await fetch("https://api.spicrawl.com/v1/scrape", { method: "POST", headers: { Authorization: `Bearer ${process.env.SPICRAWL_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ url: "https://example.com/products/42", js_render: true, proxy: process.env.MY_PROXY_URL, max_cost: 3 }), }); if (!r.ok) { const problem = await r.json(); console.log(problem.code, problem.retryable, problem.detail); } else { console.log(r.headers.get("X-Engine"), r.headers.get("X-Target-Status")); } ``` ```bash title="CLI" spicrawl scrape https://example.com/products/42 --js --proxy "$MY_PROXY_URL" --max-cost 3 --retry 3 ``` ## What a challenge looks like HTTP 502, `application/problem+json`, 0 credits: ```json { "type": "https://docs.spicrawl.com/errors#UPSTREAM_CHALLENGE", "title": "Target served a bot challenge", "status": 502, "code": "ERR::UPSTREAM::CHALLENGE", "detail": "Cloudflare served a bot challenge instead of the page, so no content could be fetched (...). Nothing was charged.", "retryable": true, "target_status": 403, "request_id": "01JAX3K6Q8V1T7M2C9D4E5F6G7", "diagnostics": { "failed_at": "fetch", "engine": "fetch", "proxy": { "source": "direct" }, "hint": "Every engine tier was served a bot challenge rather than the page. Nothing was charged. The verdict is per exit and per moment, so a retry often passes." } } ``` A request fails with `ERR::UPSTREAM::CHALLENGE` when the response carries a marker that names the vendor (for example Cloudflare's `cf-mitigated` header or DataDome's `captcha-delivery.com` script). Under `mode=auto`, any detected block escalates to the next rung, and the error is returned only when every rung was challenged. On your own `proxy` or a `premium_proxy` exit, only a challenge that names its vendor escalates; see [Automatic escalation](#automatic-escalation-coming-soon). ### A bare 403 or 429 is the target's answer A 403, 429 or 503 with no vendor marker is not reported as a challenge: it looks the same as a login wall, a geo-block or a down origin. You get HTTP 200 with the site's own body, `X-Target-Status: 403` (or 429, 503) and `X-Credits-Charged: 0`. Treat it as the site's decision. A 429 from the target means you are sending that site too much traffic; slow down rather than climbing the ladder. ## Automatic escalation (coming soon) When the page comes back as a challenge that names its vendor (Cloudflare, DataDome, PerimeterX or Akamai), Spicrawl tries to get past it before returning an error: 1. **A different exit.** A plain fetch through a [managed pool](https://docs.spicrawl.com/guides/proxies-and-geo.md#managed-proxy-pool-coming-soon) exit is retried once on another exit, free. It is skipped for `sticky_key` and `session_id`, which fix the exit, for your own `proxy`, and for methods other than `GET`, `HEAD` and `OPTIONS`. 2. **The stealth browser.** A request that goes out through your own `proxy` or a `premium_proxy` exit, pins no `engine` and does not use `mode=auto` moves to Camoufox if the page is still a challenge. Camoufox waits out the interstitial or clicks its checkbox before capturing the page. You pay for the rung that returned the page: 25 credits if Camoufox did, the fetch or obscura price if an earlier rung did, and 0 if every rung was challenged (`ERR::UPSTREAM::CHALLENGE`). The request does not move to Camoufox on a transport error, on a refusal with no vendor marker (a bare `403`), or on a non-billable target status such as `404`: those come back as they would without it. The move to Camoufox is off when: * `max_cost` is below 25; * `method` is not `GET`; * the request sets `headless=false`, `response_format=pdf`, a screenshot or `actions`; * your `proxy` is not an `http://` URL (the browser can only use an `http://` proxy); * the deployment does not run Camoufox. Before the request runs, the 25 credits are reserved against your monthly allowance. If the allowance covers the first rung but not 25, the request runs without the move to Camoufox. The response says when it escalated: `X-Engine` names the engine that served, each abandoned rung adds an `X-Warning: ESCALATED` with its reason, and `diagnostics.attempts` (in an error body, or the JSON envelope after a successful climb) lists every attempt, including a retry on a different exit. Batch items never escalate this way, because a batch prices each item when it is accepted. ## Why datacenter IPs get challenged Anti-bot vendors keep reputation lists of datacenter IP ranges and challenge them on sight, however well the browser behaves. That is why rendering alone often still fails, and why rung 3 usually matters most. Supplying your own residential proxy through `proxy` works at no proxy surcharge; see [Proxies and geo](https://docs.spicrawl.com/guides/proxies-and-geo.md). ## Retry before you climb `ERR::UPSTREAM::CHALLENGE` is `retryable: true`. The verdict is per exit and per moment, so one or two retries on the same rung are free when they fail and often pass. Use `--retry N` on the CLI. ## Reading blocks in your request logs A request that failed with `ERR::UPSTREAM::CHALLENGE` is logged with `status: "blocked"`, `credits_micro: 0` and a `blocked` object: ```json { "id": "01JAX3K6Q8V1T7M2C9D4E5F6G7", "status": "blocked", "error_code": "ERR::UPSTREAM::CHALLENGE", "http_status": 403, "engine": "fetch", "proxy": { "source": "direct" }, "blocked": { "vendor": "cloudflare", "signal": "cf-mitigated response header", "rule": "decisive-signal" }, "credits_micro": 0 } ``` `vendor` is one of `cloudflare`, `datadome`, `akamai`, `perimeterx` or `generic`. `signal` names the observable marker, so you can confirm it in the response yourself. `rule` is the detection path: `decisive-signal` (a vendor-specific interstitial marker), `phrase-and-status` (challenge wording on a 403/429/503), `phrase-and-vendor` (challenge wording plus a vendor marker in a page under 8 KiB), or `empty-refusal` (a 403/429/503 with almost no visible text). List blocks with `GET /v1/requests?status=blocked` or `spicrawl logs --status blocked`, and open one with `spicrawl logs get `. ## Failure modes | Code | HTTP | What to do | | ------------------------------- | ---- | ----------------------------------------------------------------------------- | | `ERR::UPSTREAM::CHALLENGE` | 502 | Retry once or twice, then climb one rung. | | `ERR::ENGINE::UNAVAILABLE` | 503 | The rendering engine is busy. Retry after `Retry-After`. | | `ERR::LIMIT::MAX_COST_EXCEEDED` | 400 | The rung costs more than your `max_cost`. Raise it or stay on a cheaper rung. | ## Cost Challenges, timeouts and non-billable target statuses cost 0. A success is billed at the engine that produced it: 1 for fetch, 3 for obscura, 8 for chromium, and 25 when an [automatic escalation](#automatic-escalation-coming-soon) reached Camoufox (coming soon). Your own proxy adds nothing. `impersonate` is on by default and free. See [Credits](https://docs.spicrawl.com/credits.md). ## Related * [Proxies and geo](https://docs.spicrawl.com/guides/proxies-and-geo.md) * [JavaScript rendering](https://docs.spicrawl.com/guides/javascript-rendering.md) * [Sessions and logins](https://docs.spicrawl.com/guides/sessions-and-logins.md) to keep cookies once you are through * [Errors](https://docs.spicrawl.com/errors.md)