# Capture the page's own API calls > Record the XHR and fetch responses a page makes while it renders with network_capture, and read their JSON bodies from the network array. Source: https://docs.spicrawl.com/guides/network-capture Use this when the data you want arrives in the page's own API calls: a product grid loaded from `/api/search`, prices from a GraphQL endpoint, reviews fetched on scroll. That JSON is usually cleaner and more complete than anything you can pull out of the rendered DOM, and it survives redesigns that break selectors. `network_capture` records those responses while the browser renders the page and returns them under `network`. It needs a browser engine and adds no credits. ## Minimal request ```bash title="curl" curl https://api.spicrawl.com/v1/scrape \ -H "Authorization: Bearer $SPICRAWL_API_KEY" \ -H "Content-Type: application/json" \ -d '{ "url": "https://example.com/search?q=desk", "js_render": true, "network_capture": {"urls": ["/api/search"]} }' ``` ```python title="Python" import os, requests r = requests.post( "https://api.spicrawl.com/v1/scrape", headers={"Authorization": f"Bearer {os.environ['SPICRAWL_API_KEY']}"}, json={ "url": "https://example.com/search?q=desk", "js_render": True, "network_capture": {"urls": ["/api/search"]}, }, timeout=180, ) r.raise_for_status() for resp in r.json().get("network", []): if resp["status"] == 200 and "body" in resp: print(resp["url"], resp["body"]) ``` ```typescript title="TypeScript" const r = await fetch("https://api.spicrawl.com/v1/scrape", { method: "POST", headers: { Authorization: `Bearer ${process.env.SPICRAWL_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify({ url: "https://example.com/search?q=desk", js_render: true, network_capture: { urls: ["/api/search"] }, }), }); if (!r.ok) throw new Error(JSON.stringify(await r.json())); const { network = [] } = await r.json(); const results = network.filter((n: any) => n.status === 200 && "body" in n).map((n: any) => n.body); ``` ```bash title="CLI" spicrawl scrape "https://example.com/search?q=desk" --render \ --network-capture '{"urls":["/api/search"]}' ``` ## What comes back The JSON envelope, with one entry per recorded response under `network`: ```json { "url": "https://example.com/search?q=desk", "status": 200, "content": "...", "credits": 3, "engine": "obscura", "proxy_source": "direct", "warnings": [], "network": [ { "url": "https://example.com/api/search?q=desk&page=1", "status": 200, "resource_type": "fetch", "content_type": "application/json", "duration_ms": 142, "from_cache": false, "body": { "total": 38, "items": [{ "id": 42, "name": "Walnut Desk", "price": 349.0 }] } }, { "url": "https://example.com/api/search/facets?q=desk", "status": 200, "resource_type": "xhr", "content_type": "application/json", "duration_ms": 97, "body_truncated": true, "body_base64": "eyJmYWNldHMiOlt7Im5hbWUiOiJicmFuZCIs...", "encoding": "base64" } ] } ``` How to read each entry: * `body` holds the parsed JSON, present only when the response was complete, valid JSON with a JSON content type. * `body_base64` (with `encoding: "base64"`) holds any other body: HTML, text, images, or JSON that was cut off. * `body_truncated: true` means the body hit `max_body_bytes`. Truncated JSON is always delivered as base64 because it no longer parses. Raise `max_body_bytes` to get it whole. * `status` is the status of that sub-request, not of the page. The page's own status is the top-level `status`. `network_capture` forces the envelope. If you also set `response_format=markdown`, `content` holds the markdown and the response carries `X-Warning: FORMAT_COERCED`. ## Options All fields of `network_capture` are optional; `{}` records every XHR and fetch response. | Field | Default | Limits | Effect | | ---------------- | ------------------ | -------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `urls` | `[]` (everything) | strings | A response is kept when its URL contains one of these substrings. A value starting with `^` is an anchored regular expression, e.g. `"^https://api\\.example\\.com/v2/"`. | | `resource_types` | `["xhr", "fetch"]` | `document`, `stylesheet`, `image`, `media`, `font`, `script`, `xhr`, `fetch`, `websocket`, `other` | Types to record. Exact singular names; no aliases. An unknown value is 400. | | `include_bodies` | `true` | boolean | `false` returns only `url`, `status`, `resource_type` and timing: far smaller when you only need to discover endpoints. | | `max_body_bytes` | `1048576` (1 MiB) | 0–2097152 (2 MiB); 0 means default | Per-response body cap. | | `max_responses` | `100` | 0–100; 0 means default | Maximum responses recorded. | Capturing `image` or `font` responses lifts the default image and font blocking for that render, so those requests are actually made. ### Find the endpoint first When you do not know which call carries the data, capture without bodies, then narrow: ```json { "url": "https://example.com/search?q=desk", "js_render": true, "network_capture": { "include_bodies": false } } ``` Pick the URL whose name looks like the data, then rerun with `urls` set to it and bodies on. Some calls fire only after interaction; add `wait_for` or [browser actions](https://docs.spicrawl.com/guides/browser-actions.md) (for example a scroll) to trigger them before capture. ## Requires a render `network_capture` is browser-only. Set `js_render: true` or a browser `engine` (`obscura`, `chromium`). On the fetch engine there is no page making requests, and the request fails with 400 `ERR::REQUEST::INCOMPATIBLE_FLAGS` before anything is spent. ## Failure modes | Code | HTTP | What to do | | ---------------------------------- | ---- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `ERR::REQUEST::INCOMPATIBLE_FLAGS` | 400 | No browser engine. Add `js_render: true`. | | `ERR::REQUEST::INVALID_PARAMETER` | 400 | An unknown `resource_types` value, `max_body_bytes` above 2097152, `max_responses` above 100, or an unknown key inside `network_capture`. | | `ERR::ENGINE::RENDER_FAILED` | 502 | The browser could not render the page. Retryable. | | `ERR::UPSTREAM::TIMEOUT` | 504 | The render ran out of time. Retry, or narrow `wait_for`. | An empty `network` array is not an error: the page made no matching calls. Widen `urls`, add `fetch`/`xhr` to `resource_types`, or trigger the calls with `wait_for` or actions. ## Cost `network_capture` adds 0 credits. You pay the render: 3 on obscura, 8 on chromium. Failures cost 0. See [Credits](https://docs.spicrawl.com/credits.md). ## Related * [JavaScript rendering](https://docs.spicrawl.com/guides/javascript-rendering.md) * [Browser actions](https://docs.spicrawl.com/guides/browser-actions.md) to trigger lazy calls * [Structured data](https://docs.spicrawl.com/guides/structured-data.md) when the data is in the HTML