Scrape thousands of URLs with a batch job

Use this when you have a list of URLs to fetch and do not need each answer immediately: a nightly catalogue refresh, a site archive, a price sweep. You submit the list once, get a job id back with 202, and the platform works through it with its own retries while you poll. Requires an API key with the batch scope.

Minimal request

curl https://api.spicrawl.com/v1/batch \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "catalog-2026-09-22",
    "urls": ["https://example.com/products/1", "https://example.com/products/2"],
    "js_render": true
  }'

urls gives every URL the job-level settings. items gives each URL its own object with overrides and an external_id you can correlate on: {"url": "https://example.com/products/2", "external_id": "sku-42", "js_render": true}. Send one or the other, never both (400 ERR::REQUEST::INCOMPATIBLE_FLAGS). An item's value wins over the job-level value, including an explicit false. The CLI reads a URL-per-line file, a .json array of items, or a .jsonl file of items.

What a batch item can do

The worker runs each item with one fetch or one render, then returns the document in response_format. A /v1/scrape parameter is either honoured on a batch item or refused at submission, never accepted and skipped.

Refused with 400 ERR::REQUEST::INVALID_PARAMETER, which names every offending parameter: actions, screenshot and its options, network_capture, extract, extract_preset, ai_extract, autoparse, links, session_id, cache/cache_ttl, original_status, allowed_status_codes, parse_pdf: true, mode: "auto", solve_captcha: true (not available on batch jobs yet), response_format: "pdf", and a non-GET method with js_render or stealth. A screenshot or PDF is always refused, because a batch item cannot use full-browser rendering; use /v1/scrape. Nothing is queued or charged. A value that asks for nothing (screenshot: false, []) is accepted. When the parameter comes from your project's defaults rather than the request, the detail says so.

A markdown or text item whose document is not HTML (a PDF, an image, JSON) fails with ERR::EXTRACTION::FAILED: the batch worker does not parse PDFs. Request html to get the document as retrieved.

For extraction, screenshots or actions on many URLs, run /v1/scrape with client-side concurrency and add the flags you need:

spicrawl scrape - --concurrency 8 --extract @fields.json < urls.txt > pages.jsonl

What comes back

Submission returns 202 with the job object: id, status: "queued", status_url, results_url, total_items, estimated_credits (the hold) and progress. Read warnings: it reports a clamped concurrency, an ignored engine or headless field, or items not yet handed to the queue. Do not resubmit in that case; a recovery sweep dispatches them, and a resubmission is a second job billed again.

Poll GET /v1/batch/{id} every 2–5 seconds for jobs under about 1,000 items, every 10–30 seconds for larger ones. Progress counters are cheap to read.

GET /v1/batch/{id}/results returns finished items as JSON Lines (application/x-ndjson), one object per line, in seq order:

{"seq":1,"external_id":"sku-42","url":"https://example.com/products/2","status":"succeeded","attempts":1,"http_status":200,"request_id":"01J9Z7A1B2C3D4E5F6G7H8J9KA","credits_micro":4000000,"bytes":326000,"duration_ms":1840,"result":{"content":"<!doctype html>…","bytes":326000},"finished_at":"2026-09-22T06:41:14Z"}

• status is succeeded, failed (read error.code and error.retryable), cancelled or skipped. Unfinished items are never listed.
• http_status is the target's status. A 404 here is a successful scrape of a missing page.
• result.content is the page as plain text in the item's response_format: the HTML, markdown or text itself. Your JSON parser has already decoded the line, so use the string as it is and do not json.loads it again. For response_format: "json" on batch it is the document as retrieved, the same string as html, not the /v1/scrape envelope (see the table above). result.bytes is the length of content.
• GET /v1/batch/{id}/tasks/{seq}/content is different: it returns the payload as a single JSON string literal, which a raw HTTP client decodes once.
• result is absent while the job is still running: bodies are inlined from a file written when the job becomes terminal.
• Paging is in headers. If X-Next-Cursor is present, request again with cursor=<value> (the Link header has the full URL). limit is 1–5,000, default 500. Filter with status=failed to list only failures.

Body caps: 512 KiB per item and 2 MiB per page, spent in seq order. Exactly one of these applies to each result:

Results are readable until results_expire_at, 72 hours after submission by default. After that the results endpoint answers 410 ERR::REQUEST::BEYOND_RETENTION; the job object stays readable.

Limits and options

• Body up to 1 MiB (413), 10,000 items per call, job-level parameters up to 64 KiB serialised, each item's overrides up to 8 KiB.
• A bad item fails the whole submission; detail starts with Item N:.
• open: true keeps the job accepting items via POST /v1/batch/{id}/items (same body shape, 10,000 per call, 100,000 per job). An open job never completes on its own: call POST /v1/batch/{id}/close when done. Appended items use the job's original settings plus their own overrides.

Retry and cancel

• POST /v1/batch/{id}/retry resets every failed item to queued. Succeeded, cancelled and skipped items never re-run, and attempts is not reset. A terminal job goes back to queued. With nothing failed, it returns items_reset: 0.
• POST /v1/batch/{id}/cancel moves a live job to cancelling; in-flight items finish (and are charged if they succeed), then the job becomes cancelled. Cancelling a terminal job is 409.

CLI: spicrawl batch retry <id>, spicrawl batch cancel <id>, spicrawl batch append <id> more.txt, spicrawl batch close <id>, spicrawl batch wait <id> --max-wait 30m (exit 11 on timeout).

Failure modes

Per-item failures are on the result line in error.code, from the same catalogue as /v1/scrape (ERR::UPSTREAM::TIMEOUT, ERR::PROXY::EXHAUSTED, ...). Re-run retryable ones with retry. ERR::EXTRACTION::FAILED on a markdown or text item means the document was not HTML; it is permanent, so resubmit that URL with response_format: "html".

Cost

Submitting is free (X-Credits-Charged: 0) but reserves the dearest possible cost of every item against your monthly allowance; that hold is estimated_credits, and the response's X-Credits-Remaining is the allowance left after it. Append and retry responses carry X-Credits-Remaining after their holds too. Credits held for items that fail, are cancelled or skipped are returned to the monthly allowance within about a minute of the item finishing. Appending to an open job reserves the appended items the same way, and is refused with 402 if the allowance cannot cover them. Each item is charged on success only, at the /v1/scrape price for how it was fetched (1 credit for a plain fetch, 3 with js_render). Billable target statuses are 200, 404 and 410. A js_render item whose page renders empty (no visible content) is not charged; its result carries "warnings":["EMPTY_RENDER"]. Retrying a failed item (retry) holds its price again first and is refused with 402if the allowance cannot cover it. Actual spend isprogress.credits_charged`. See Credits.

Related

• CLI batch commands
• Markdown for LLMs for per-URL markdown with /v1/scrape
• Proxies and geo
• Rate limits