Submit a batch job

Queues many URLs as one asynchronous job and returns `202` with the job object immediately.

Requires scope: batch

Queues many URLs as one asynchronous job and returns 202 with the job object immediately. Supply either urls (same settings for every URL) or items (one object per URL, with per-item overrides), never both. Every /v1/scrape parameter at the top level (except url) is a job-wide default; an item's own value wins over it, including an explicit false.

Order of work: validate and route every item (a bad item N fails the whole submission with Item N: in detail), reserve the dearest possible cost of the whole job against the monthly credit allowance, write the manifest, enqueue. Nothing is charged at submission (X-Credits-Charged: 0); items are charged individually, only on success, by the worker.

Next step: poll GET /v1/batch/{batchID} (the status_url) until status is completed, failed or cancelled, then page GET /v1/batch/{batchID}/results. Results are readable while the job runs, too.

Not idempotent: there is no Idempotency-Key. Resubmitting the same body creates a second job that is scraped and billed again. If a 202 carries a warning saying items were not all dispatched, do NOT resubmit — the worker's recovery sweep dispatches them.

Limits: request body 1 MiB (1,048,576 bytes); at most 10,000 items per call; job-level parameters at most 65,536 bytes serialised; each item's overrides plus external_id at most 8,192 bytes serialised.

Authorization

bearerAuth

AuthorizationBearer <token>

Authorization: Bearer <key>. Read the key from the SPICRAWL_API_KEY environment variable; never hard-code or log it. spicrawl_test_… keys can never spend live credits. Scopes: scrape, batch, sessions (granted by default), browser and read (granted deliberately). A missing scope is 403 ERR::AUTH::INSUFFICIENT_SCOPE naming the scope.

In: header

Request Body

application/json

TypeScript Definitions

Use the request body type in TypeScript.

Body of POST /v1/batch (and of POST /v1/batch/{batchID}/items). Exactly one of urls or items must be non-empty. Top-level BatchScrapeParams fields are job-wide defaults. A top-level url is rejected with 400 — put targets in urls or items[].url. Unknown fields are a 400. Must be ≤ 1 MiB; ≤ 10,000 targets.

method?method

Validated as on /v1/scrape, but the batch worker always sends GET. Leave unset.

Default"GET"

Value in

"GET"
"POST"
"PUT"
"PATCH"
"DELETE"
"HEAD"
"OPTIONS"

js_render?js_render

Render the page with the obscura browser engine. Honoured by the batch worker. Default false (or the project default).

Defaultfalse

stealth?stealth

Coming soon Render with the camoufox anti-detection engine; takes precedence over js_render. Honoured by the batch worker. Default false (or the project default).

Defaultfalse

impersonate?impersonate

Applied by the batch worker on fetched items. Default true, as on /v1/scrape; false on the item, the job or the project defaults opts out.

Defaulttrue

mode?"auto"

Validated and priced as on /v1/scrape (auto reserves the dearest tier). The batch worker does not escalate; it runs fetch unless js_render or stealth is set. Cannot be combined with engine (400 INCOMPATIBLE_FLAGS).

Value in

"auto"

engine?engine

Engine camoufox is coming soon. Engine pin. On batch only fetch, obscura, camoufox are allowed; chromium is rejected with 400 INVALID_PARAMETER (use /v1/scrape for chromium). Also 400 INCOMPATIBLE_FLAGS for: mode=auto plus any pin; stealth=true with a pin other than camoufox; js_render=true with fetch; screenshot=true with fetch or obscura. A pin the plan does not include is 403 (plans whose entitlements cannot be read allow only fetch and obscura). To be sure the worker renders, set js_render (obscura) or stealth (camoufox) rather than relying on the pin alone.

Value in

"fetch"
"obscura"
"camoufox"
"chromium"

premium_proxy?premium_proxy

Coming soon Use the residential proxy pool instead of datacenter. Honoured by the batch worker; ignored when proxy is set.

Defaultfalse

proxy_country?proxy_country

Coming soon Exit country for pool proxies. Honoured by the batch worker.

Match^[A-Za-z]{2}$

proxy?proxy

Your own proxy URL; wins over the pool flags. Honoured by the batch worker (without proxy verification).

proxy_verify?proxy_verify

Validated as on /v1/scrape; the batch worker always sends the custom proxy with verification off.

Defaultfalse

session_id?session_id

Validated as on /v1/scrape; batch items do not run inside a session. Not applied.

sticky_key?sticky_key

Coming soon Validated as on /v1/scrape; not applied by the batch worker.

wait?wait

Validated as on /v1/scrape; not applied by the batch worker.

Range0 <= value <= 30000

Default0

wait_for?wait_for

Validated as on /v1/scrape; not applied by the batch worker.

wait_for_timeout?wait_for_timeout

Validated as on /v1/scrape; not applied by the batch worker.

Range0 <= value <= 120000

Default0

actions?array<||||||>

Validated as on /v1/scrape; the batch worker does not run browser actions.

Itemsitems <= 50

block_resources?array<>

Resource classes to block while rendering. Honoured on rendered items only; unset blocks images and fonts, none blocks nothing, a list replaces the default. Falls back to the project default when unset.

headless?headless

Validated as on /v1/scrape (the project default for headless is NOT merged on batch). false needs chromium, which batch cannot pin. Not applied by the batch worker.

screenshot?screenshot

Validated and priced as on /v1/scrape; the batch worker returns no screenshot. With engine fetch/obscura it is a 400.

Defaultfalse

screenshot_fullpage?screenshot_fullpage

Validated as on /v1/scrape; not applied by the batch worker.

Defaultfalse

screenshot_selector?screenshot_selector

Validated as on /v1/scrape; not applied by the batch worker.

screenshot_format?screenshot_format

Validated as on /v1/scrape; not applied by the batch worker.

Value in

"png"
"jpeg"
"jpg"
"webp"

screenshot_quality?screenshot_quality

Validated as on /v1/scrape; not applied by the batch worker.

Range0 <= value <= 100

Default0

custom_headers?

Validated as on /v1/scrape; the batch worker does not send custom headers.

autoparse?autoparse

Validated as on /v1/scrape; not applied by the batch worker.

Defaultfalse

links?links

Validated as on /v1/scrape; not applied by the batch worker.

Defaultfalse

network_capture?

Validated as on /v1/scrape (see NetworkCapture); not applied by the batch worker.

extract?|

Selector map or JSON Schema form. An empty object is 400. Refused on PDF targets (502 ERR::EXTRACT::FAILED).

extract_preset?extract_preset

Validated as on /v1/scrape; not applied by the batch worker.

ai_extract?|

Exactly one of prompt or schema (both is 400 INCOMPATIBLE_FLAGS, neither is 400 INVALID_PARAMETER). The model reads the page as cleaned markdown; output lands in data. +4 credits.

main_content_only?main_content_only

Validated as on /v1/scrape; not applied by the batch worker.

include_tags?array<string>

Validated as on /v1/scrape; not applied by the batch worker.

exclude_tags?array<string>

Validated as on /v1/scrape; not applied by the batch worker.

parse_pdf?parse_pdf

Validated as on /v1/scrape (default true); not applied by the batch worker.

Defaulttrue

response_format?response_format

Validated as on /v1/scrape, but the batch worker always returns the raw page body regardless of this value (no markdown, json, text or pdf conversion).

Default"html"

Value in

"html"
"markdown"
"text"
"json"
"pdf"

cache?cache

Validated as on /v1/scrape, but on batch it defaults to false (not true) and the worker never reads from the cache.

Defaulttrue

cache_ttl?cache_ttl

Validated as on /v1/scrape; 0 does not imply cache=false on batch. Not applied by the batch worker.

Range0 <= value

Default172800

max_cost?max_cost

Per-ITEM credit ceiling, checked at submission against the item's dearest outcome (400 if exceeded). For a ceiling on the whole run use credit_budget.

Range0 <= value

Default0

original_status?original_status

Validated as on /v1/scrape; not applied (result lines always carry the target status in http_status).

Defaultfalse

allowed_status_codes?array<>

Validated as on /v1/scrape; not applied. Billing is fixed to target statuses 200, 404 and 410.

name?string

Free-text label shown on the job object and in the dashboard. No uniqueness; omit for none.

urls?array<>

Shorthand list of targets that all use the job-level settings. Mutually exclusive with items (both → 400 INCOMPATIBLE_FLAGS; neither → 400 MISSING_PARAMETER). Order defines seq (0-based).

Items1 <= items <= 10000

items?array<>

Long form, one object per target with optional per-item overrides and external_id. Mutually exclusive with urls. Order defines seq (0-based).

Items1 <= items <= 10000

concurrency?integer

Max items of this job in flight at once. Values below 1 become 1; values above 50 are clamped to 50 and reported in the job's warnings (not an error).

Default10

priority?integer

Scheduling priority relative to your other jobs. Outside 0-1000 is a 400.

Range0 <= value <= 1000

Default100

max_attempts?integer

Attempts per item before it is marked failed (only retryable failures are re-attempted). Outside 1-10 is a 400.

Range1 <= value <= 10

Default3

failure_threshold?integer

Abort the job as failed once this many items have failed. Omit to never abort early. 0 or negative is a 400.

Formatint64

Range1 <= value

credit_budget?integer

Ceiling on the WHOLE run in whole credits; the worker stops charging work beyond it. Omit to use the job's projected cost; an open job then has no ceiling, because its projection covers only the items it was submitted with. Stored as given in params.credit_budget_micro. 0 or negative is a 400. Different from max_cost, which is per item.

Range1 <= value

webhook_endpoint_id?string

Id of a webhook endpoint registered for this organization; it receives the job object when the job finishes (notify-only — fetch results yourself). A value that is not a valid id is a 400; blank is treated as absent.

open?boolean

Keep the job accepting more items via POST /v1/batch/{batchID}/items. An open job never completes on its own — call POST /v1/batch/{batchID}/close when done. Total across appends is capped at 100,000 items.

Defaultfalse

Response Body

`application/json`

`application/problem+json`

curl -X POST "https://example.com/v1/batch" \  -H "Content-Type: application/json" \  -d '{    "name": "nightly-catalog",    "urls": [      "https://example.com/products/1",      "https://example.com/products/2"    ],    "js_render": true,    "premium_proxy": true,    "proxy_country": "us",    "concurrency": 20  }'

{  "id": "01J9Z6T3W9E21T5TZARVJRVN5C",  "project_id": "01J8X2M4C7Q1H5V9P3R6T8W0YB",  "name": "catalog-2026-09-22",  "status": "running",  "status_url": "/v1/batch/01J9Z6T3W9E21T5TZARVJRVN5C",  "results_url": "/v1/batch/01J9Z6T3W9E21T5TZARVJRVN5C/results",  "params": {    "js_render": true,    "proxy_country": "us",    "credit_budget_micro": 8000000  },  "total_items": 2,  "concurrency": 20,  "priority": 100,  "max_attempts": 3,  "estimated_credits": 8,  "estimated_credits_micro": 8000000,  "progress": {    "total": 2,    "completed": 1,    "succeeded": 1,    "failed": 0,    "cancelled": 0,    "skipped": 0,    "remaining": 1,    "percent_complete": 50,    "credits_charged": 4,    "credits_charged_micro": 4000000,    "bytes": 326000  },  "submitted_at": "2026-09-22T06:41:12Z",  "started_at": "2026-09-22T06:41:13Z",  "results_expire_at": "2026-09-25T06:41:12Z"}

Submit a batch job

Authorization

Request Body

Response Body

202application/json

400application/problem+json

401application/problem+json

402application/problem+json

403application/problem+json

413application/problem+json

500application/problem+json

503application/problem+json

`application/json`

`application/problem+json`

`application/problem+json`

`application/problem+json`

`application/problem+json`

`application/problem+json`

`application/problem+json`

`application/problem+json`