Render JavaScript pages

Use this when the page you fetched is an empty shell: a <div id="root"></div>, a "Loading…" placeholder, or markdown with no article in it. A plain fetch, the default, downloads the HTML and runs no JavaScript. js_render=true loads the page in a browser, lets its scripts build the content, then captures it. Rendering is also required for wait, wait_for, actions, block_resources, network_capture, screenshot and response_format=pdf; setting any of those without rendering is a 400, never silently ignored.

Minimal request

curl https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products/42",
    "js_render": true,
    "wait_for": ".price",
    "response_format": "markdown"
  }'

What comes back

The rendered page, in the format you asked for. Check the headers:

HTTP/1.1 200 OK
Content-Type: text/markdown; charset=utf-8
X-Target-Status: 200
X-Credits-Charged: 3

• X-Credits-Charged is the price of the rendering tier that served the page, and X-Engine names it: fetch, js (JavaScript rendering) or browser (full-browser rendering).
• X-Warning appears when something was adjusted. RENDER_DEGRADED means wait_for never matched and you got the page as it stood. EMPTY_RENDER means the page rendered with no visible content: nothing is charged, so retry with wait_for or a longer wait. ENGINE_SUBSTITUTED means this deployment served the request on a different rendering tier than usual (see below). ENGINE_IGNORED means the request sent engine or headless, which never change anything (see below).

How rendering is chosen

You do not choose how a page is rendered. Spicrawl picks the path from what the request needs, and you are billed for the tier that served it.

On a premium_proxy Coming soon exit these cost 10, 25 and 32. stealth=true Coming soon renders with a hardened fingerprint on full-browser rendering and is billed for the tier that served; see Anti-bot.

The engine and headless fields are deprecated and ignored: they never change how a page is rendered, nothing returns a 400, and the response carries X-Warning: ENGINE_IGNORED (in the JSON envelope, under warnings). The CLI flags --engine and --headless map to those fields and are ignored too.

Full-browser rendering always runs with a real display and adds about 1 s of startup.

ENGINE_SUBSTITUTED

js_render=true asks for JavaScript rendering, not for a specific path. If this deployment does not run the usual one, the request is served by the next tier up that it does run, which can cost more (8 or 25 credits), and is flagged with X-Warning: ENGINE_SUBSTITUTED: the requested rendering was unavailable; served by browser rendering. (or JavaScript rendering), and X-Engine names the tier that ran. Set max_cost if you would rather fail with ERR::LIMIT::MAX_COST_EXCEEDED than pay more.

Options that matter

Without wait or wait_for, a render waits a few seconds after load for the page's content to appear, and keeps waiting (up to that limit) while the page still shows a loading placeholder. Use wait_for when you need a specific element.

block_resources defaults

When you omit block_resources, full-browser rendering blocks nothing, so images and fonts load. They are most of a page's bytes, so a full-browser pass transfers more than it needs to if you only want the text: send ["images", "fonts"] to skip them. An explicit list blocks exactly the classes you name; ["none"] blocks nothing. JavaScript rendering cannot block resources, so a request with an explicit list runs on full-browser rendering, and no default block is applied to JavaScript rendering. Block script only when the content is already in the HTML, or the page will not build.

mode=auto

mode=auto tries a plain fetch first, then JavaScript rendering, then full-browser rendering, then stealth rendering Coming soon. It moves up a rung only when the previous one was met by a bot challenge or a target status that is not billable. You pay only for the rung that succeeded, and 0 if all fail. If full-browser rendering is refused by a challenge that names its vendor, the request ends there with ERR::UPSTREAM::CHALLENGE instead of trying stealth rendering Coming soon; see Anti-bot. Two things to know:

• max_cost and your monthly allowance are checked against the price of the ladder's dearest rung (25 credits for stealth rendering; 32 for full-browser rendering on a premium_proxy Coming soon exit), even when the request stops at fetch.
• When the request escalated, the response carries one X-Warning: ESCALATED per abandoned rung (for example ESCALATED: fetch (1 credit): blocked by challenge page (Cloudflare …)), and the JSON envelope includes diagnostics.attempts listing each rung and why it was abandoned.

mode=auto cannot be combined with js_render, stealth or solve_captcha (400 ERR::REQUEST::INCOMPATIBLE_FLAGS). It does reach full-browser rendering, when JavaScript rendering was met by a challenge.

Failure modes

All of these cost 0 credits.

Cost

Per successful request: plain fetch 1, JavaScript rendering 3, full-browser rendering 8, stealth rendering Coming soon 25. On a premium_proxy Coming soon exit: 10, 25, 32 and 25. wait, wait_for and block_resources add nothing. Failures cost 0. Read X-Credits-Charged for the amount billed. See Credits.

Related

• Anti-bot when a render returns a challenge page
• Browser actions to click, scroll and fill before capture
• Network capture to read the page's own JSON API calls
• Screenshots and PDF