Use your own proxy

Route requests through your own proxy with proxy and proxy_verify, and read how a request was routed. Spicrawl's managed proxy pool is coming soon.

Use this when a site blocks Spicrawl's IPs, shows different prices or content per country, or requires every request in a flow to come from the same IP. Route the request through a proxy you control with proxy; the site then sees your proxy's IP and country.

Coming soon

Spicrawl's managed proxy pool (datacenter and residential exits, premium_proxy, country selection, sticky exits) is not available yet. Bring your own proxy as shown below.

Minimal request

Fetch a page through your own proxy:

curl -i https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com/products/42", "proxy": "http://user:pass@proxy.example.net:8000", "proxy_verify": true}'

What comes back

The page, plus a header that says how it was routed:

HTTP/1.1 200 OK
X-Engine: fetch
X-Proxy-Source: custom
X-Target-Status: 200
X-Credits-Charged: 1

X-Proxy-Source is custom (your proxy) or direct (no proxy).
In the JSON envelope the same fact is proxy_source.

Options that matter

Field	Default	Effect
`proxy`	none	Your own proxy: `http://`, `https://`, `socks5://` or `socks5h://`, credentials in the userinfo (`http://user:pass@proxy.example.net:8000`). Adds 0 proxy credits.
`proxy_verify`	`false`	Probe your `proxy` before the engine runs, so a dead proxy fails fast. Requires `proxy`.

To pick a country, use a proxy that exits in that country. To keep one IP across a flow, use a proxy that gives you a fixed exit, and a session for cookies.

With your own proxy, requests are priced as the direct rate: fetch 1, obscura 3, chromium 8. Loopback hosts are refused with ERR::SECURITY::SSRF_BLOCKED.

Failure modes

Code	HTTP	What to do
`ERR::PROXY::UNREACHABLE`	502	Your `proxy` did not accept a connection. Check the URL; add `proxy_verify=true` to fail before the engine cost. Not retryable.
`ERR::PROXY::AUTH_FAILED`	502	Your proxy rejected the credentials in the URL. Not retryable.
`ERR::PROXY::RATE_LIMITED`	502	Your proxy is rate limited. Retryable.
`ERR::REQUEST::INCOMPATIBLE_FLAGS`	400	`proxy_verify` without `proxy`.
`ERR::REQUEST::INVALID_PARAMETER`	400	`proxy` has an unsupported scheme or no host.
`ERR::SECURITY::SSRF_BLOCKED`	400	Your `proxy` points at a loopback host.

All of these cost 0 credits.

Cost

Engine	Direct / your proxy
fetch	1
obscura (`js_render`)	3
chromium	8

proxy_verify adds nothing. Failures cost 0. See Credits.

Managed proxy pool Coming soon

Coming soon

Spicrawl-managed exits, including residential exits (premium_proxy), country selection (proxy_country) and sticky exits (sticky_key), are not available yet. Until then, supply your own proxy with proxy.

A browser render through a managed exit takes its timezone and locale from proxy_country, or, when you send none, from the country of the exit it was assigned, so the browser's clock and language match the IP the site sees. A plain fetch through a managed exit that meets a named vendor's bot challenge is retried once on another exit, free; see Anti-bot.