Use your own proxy

Route requests through your own proxy with proxy and proxy_verify, which also lets Cloudflare-protected pages clear automatically, and read how a request was routed. Spicrawl's managed proxy pool is coming soon.

Use this when a site blocks Spicrawl's IPs, shows different prices or content per country, or requires every request in a flow to come from the same IP. Route the request through a proxy you control with proxy; the site then sees your proxy's IP and country.

Coming soon

Spicrawl's managed proxy pool (datacenter and residential exits, premium_proxy, country selection, sticky exits) is not available yet. Bring your own proxy as shown below.

Minimal request

Fetch a page through your own proxy:

curl -i https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products/42", "proxy": "http://user:pass@proxy.example.net:8000", "proxy_verify": true}'

What comes back

The page, plus a header that says how it was routed:

HTTP/1.1 200 OK
X-Proxy-Source: custom
X-Target-Status: 200
X-Credits-Charged: 1

X-Proxy-Source is custom (your proxy) or direct (no proxy).
In the JSON envelope the same fact is proxy_source.

Options that matter

Field	Default	Effect
`proxy`	none	Your own proxy: `http://`, `https://`, `socks5://` or `socks5h://`, credentials in the userinfo (`http://user:pass@proxy.example.net:8000`). Adds 0 proxy credits.
`proxy_verify`	`false`	Probe your `proxy` before the engine runs, so a dead proxy fails fast. Requires `proxy`.

To pick a country, use a proxy that exits in that country. To keep one IP across a flow, use a proxy that gives you a fixed exit, and a session for cookies. When you know the exit country, also send proxy_country with its ISO code: the URL stays the exit (nothing routes on it), but a rendered request takes its timezone, locale and Accept-Language from that country instead of the UTC/en-US default — a mismatched clock and language against the exit IP is itself a signal anti-bot vendors score.

A page behind Cloudflare, or another managed challenge, that challenges a request sent through your own http:// proxy is moved to stronger rendering on the same exit automatically, and you pay only for the tier that returned the page. A residential proxy works best; see Cloudflare-protected pages and the Cloudflare guide. If a request uses your own proxy and hits a blocking challenge that Spicrawl solves, your proxy's address and credentials are shared with the third-party solving service for that solve. Without your own proxy, a blocked request may be completed through a residential exit of ours at no change in price (only the 5-credit solve surcharge applies, when a solve succeeds).

With your own proxy, requests are priced as the direct rate: plain fetch 1, JavaScript rendering 3, full-browser rendering 8. Loopback hosts are refused with ERR::SECURITY::SSRF_BLOCKED.

Failure modes

Code	HTTP	What to do
`ERR::PROXY::UNREACHABLE`	502	Your `proxy` did not accept a connection. Check the URL; add `proxy_verify=true` to fail before the engine cost. Not retryable.
`ERR::PROXY::AUTH_FAILED`	502	Your proxy answered `407`: it rejected the credentials in the URL. The detail names the proxy, never the password. Not retryable.
`ERR::PROXY::RATE_LIMITED`	502	Your proxy is rate limited. Retryable.
`ERR::REQUEST::INCOMPATIBLE_FLAGS`	400	`proxy_verify` without `proxy`.
`ERR::REQUEST::INVALID_PARAMETER`	400	`proxy` has an unsupported scheme or no host.
`ERR::SECURITY::SSRF_BLOCKED`	400	Your `proxy` points at a loopback host.

All of these cost 0 credits.

Cost

Rendering	Direct / your proxy	Premium exit Coming soon
Plain fetch	1	10
JavaScript rendering (`js_render`)	3	25
Full-browser rendering (`screenshot`, `response_format=pdf`, `actions`)	8	32

proxy_verify adds nothing. Failures cost 0. See Credits.

Managed proxy pool Coming soon

Coming soon

Spicrawl-managed exits, including residential exits (premium_proxy), country selection (proxy_country) and sticky exits (sticky_key), are not available yet. Until then, supply your own proxy with proxy.

Premium exits will enable automatic challenge handling: a page on one that shows a bot challenge, including a Cloudflare one, will be moved to a stronger rendering path on the same exit, as it already is on your own proxy; see Cloudflare-protected pages.

A browser render through a managed exit takes its timezone and locale from proxy_country, or, when you send none, from the country of the exit it was assigned, so the browser's clock and language match the IP the site sees. A plain fetch through a managed exit that meets a named vendor's bot challenge is retried once on another exit, free; see Anti-bot.