spicrawl batch

spicrawl batch runs many URLs as one server-side job (POST /v1/batch). You submit a file of URLs, poll until the job finishes, then download one JSON line per item.

spicrawl batch submit urls.txt --render --name nightly --wait --max-wait 20m
spicrawl batch results 01J9Z6V0Q8M4K2T7R3N5B1C9XA --all -o results.jsonl

When to use batch instead of scrape -

Submit a job

spicrawl batch submit <file|-> [scrape flags] [job flags]

Input formats

An item is {"url": "...", "external_id": "..."} plus any per-item scrape overrides. external_id is your correlation id; it comes back on the item's result line. An item without url exits 2.

{"url":"https://example.com/products/1","external_id":"sku-1"}
{"url":"https://example.com/products/2","external_id":"sku-2","js_render":false}

spicrawl batch submit items.jsonl --render --proxy "$MY_PROXY_URL"

Scrape flags set job-wide defaults; item fields override them. The scrape flags are the same as spicrawl scrape, except --wait (see below).

Job flags

Output

Without --wait, submit prints the job as soon as it is accepted. In JSON mode that is the API's BatchJob object unchanged:

id=$(spicrawl batch submit urls.txt --render | jq -r .id)

In human mode it prints id, status, progress, credits, submitted and a next: spicrawl batch wait <id> hint on stderr.

Wait for a job

spicrawl batch wait 01J9Z6V0Q8M4K2T7R3N5B1C9XA --max-wait 10m
spicrawl batch wait "$id" --poll 10s --json | jq '.progress'

Polls GET /v1/batch/{id} every --poll (default 5s) until the status is completed, failed or cancelled, then prints the job.

• In human mode a progress line on stderr shows batch <id>: running 412/1000 (41.2%) succeeded 405 failed 7. JSON mode and --quiet print no progress.
• A 429, or a retryable 503, does not stop the wait: the next poll waits for the server's Retry-After.
• If --max-wait (default 30m) elapses first, it prints the last job seen and exits 11 with batch <id> is still running after 10m0s; resume with: spicrawl batch wait <id>. Run the same command again to keep waiting.

spicrawl batch wait "$id" --max-wait 5m
case $? in
  0)  echo "finished" ;;
  11) echo "still running; try again later" ;;
  *)  echo "failed" >&2 ;;
esac

A job whose status is failed still exits 0 from wait: the wait itself succeeded. Check .status and .error_code in the printed job.

Inspect jobs

spicrawl batch get 01J9Z6V0Q8M4K2T7R3N5B1C9XA --json | jq '{status, progress}'
spicrawl batch list --status running
spicrawl batch list --all --json | jq -r '.batches[] | select(.status=="failed") | .id'

batch list prints {"batches": [...], "next_cursor": "..."} in JSON mode, and an ID STATUS NAME DONE SUBMITTED table otherwise. next_cursor is present only when more pages exist and you did not pass --all.

Download results

spicrawl batch results <id> [--status S] [--all] [--limit N] [--cursor C] [-o FILE]

Fetches GET /v1/batch/{id}/results: one JSON object per finished item, in seq order. In JSON mode, or with -o, the lines are written unchanged as JSONL; in human mode a SEQ STATUS HTTP URL BYTES table is printed.

Each line has seq, external_id, url, status, attempts, http_status, request_id, bytes, duration_ms, result and error. result.content is the stored page as JSON text (a JSON string holding the HTML), so decode it once:

# Everything, to a file
spicrawl batch results "$id" --all -o results.jsonl

# Only failures, with their error codes
spicrawl batch results "$id" --status failed --all | jq -r '[.seq, .url, .error.code] | @tsv'

# The HTML of every succeeded item
spicrawl batch results "$id" --status succeeded --all | jq -r '.result.content | fromjson'

Bodies are inlined only once the job is terminal. Results expire at the job's results_expire_at (72 hours after submission by default); after that the API answers ERR::REQUEST::BEYOND_RETENTION (HTTP 410, exit 4).

Change a running job

spicrawl batch retry "$id" && spicrawl batch wait "$id" --max-wait 15m
spicrawl batch cancel "$id" && spicrawl batch wait "$id" --max-wait 2m

Open jobs for crawls

id=$(spicrawl batch submit seeds.txt --open --name crawl | jq -r .id)
echo https://example.com/discovered/17 | spicrawl batch append "$id" -
spicrawl batch append "$id" more.jsonl
spicrawl batch close "$id" && spicrawl batch wait "$id"

Errors

See the batch guide for the job lifecycle and webhooks.