spicrawl batch

spicrawl batch runs many URLs as one server-side job (POST /v1/batch). You submit a file of URLs, poll until the job finishes, then download one JSON line per item.

spicrawl batch submit urls.txt --render --name nightly --wait --max-wait 20m
spicrawl batch results 01J9Z6V0Q8M4K2T7R3N5B1C9XA --all -o results.jsonl

When to use batch instead of scrape -

Submit a job

spicrawl batch submit <file|-> [scrape flags] [job flags]

Input formats

An item is {"url": "...", "external_id": "..."} plus any per-item scrape overrides. external_id is your correlation id; it comes back on the item's result line. An item without url exits 2.

{"url":"https://example.com/products/1","external_id":"sku-1"}
{"url":"https://example.com/products/2","external_id":"sku-2","js_render":false}

spicrawl batch submit items.jsonl --render --proxy "$MY_PROXY_URL"

Scrape flags set job-wide defaults; item fields override them. batch submit takes the spicrawl scrape flags whose fields a batch job runs, mapped to the same API fields: --format (html, markdown, text, json), --render, --stealth, --engine, --impersonate, --method, --body, --header, --proxy, --proxy-verify, --premium-proxy, --country, --sticky-key, --wait-for, --wait-for-timeout, --block, --headless, --main-content, --no-main-content, --include, --exclude and --max-cost. --stealth, --premium-proxy, --country and --sticky-key are Coming soon. --engine and --headless are deprecated and ignored: rendering is chosen from --render and --stealth, and the job's warnings say the field was ignored. The render delay is --render-wait (the scrape wait field), because --wait waits for the job (see below).

spicrawl batch submit urls.txt --render --render-wait 1500 --format markdown --main-content

Job flags

Output

Without --wait, submit prints the job as soon as it is accepted. In JSON mode that is the API's BatchJob object unchanged:

id=$(spicrawl batch submit urls.txt --render | jq -r .id)

In human mode it prints id, status, progress, credits, submitted and a next: spicrawl batch wait <id> hint on stderr.

Wait for a job

spicrawl batch wait 01J9Z6V0Q8M4K2T7R3N5B1C9XA --max-wait 10m
spicrawl batch wait "$id" --poll 10s --json | jq '.progress'

Polls GET /v1/batch/{id} every --poll (default 5s) until the status is completed, failed or cancelled, then prints the job.

• On a terminal a progress bar on stderr shows the count, the elapsed time and an ETA, labelled with the job id, and a ✓ or ✗ line ends the wait. Otherwise, in human mode, a line shows batch <id>: running 412/1000 (41.2%) succeeded 405 failed 7 whenever the counts change. JSON mode and --quiet print no progress.
• A 429 waits for the server's Retry-After (at most 60 seconds) before the next poll. A transient 5xx or network error is retried with backoff; five polls in a row failing that way ends the wait with that error.
• If --max-wait (default 30m) elapses first, it prints the last job seen and exits 11 with batch <id> is still running after 10m0s; resume with: spicrawl batch wait <id>. Run the same command again to keep waiting.

spicrawl batch wait "$id" --max-wait 5m
case $? in
  0)  echo "completed" ;;
  4)  echo "failed or cancelled; partial results: batch results" >&2 ;;
  11) echo "still running; try again later" ;;
  *)  echo "error" >&2 ;;
esac

Watch a job

spicrawl batch watch 01J9Z6V0Q8M4K2T7R3N5B1C9XA
spicrawl batch submit urls.txt --name nightly | jq -r .id | xargs spicrawl batch watch

batch watch <id> (0.1.4 or later) is a full-screen dashboard on a terminal: a progress bar with done, failed and total, the ETA and credits, and a table of the finished items (status, HTTP status, tier, duration, URL), newest first. The table lists up to the first 1,000 items in submission order, while the job's own counters are always complete. A tier shows once the request log has the item's request. --poll (default 2s) and --max-wait (default 30m) work as for wait.

When the job finishes, the dashboard closes, prints the final job and exits as batch wait does: 0 completed, 4 failed or cancelled, 11 if --max-wait ran out first. A failed poll does not close it: the error shows on the status bar and the poll is retried with backoff. A wrong key or an unknown job id fails at once with its usual exit code, before the screen opens. NO_COLOR turns colour off.

The dashboard needs a terminal on stdin and stdout. When either is not one, or with --json, -q, CI or TERM=dumb, there is no dashboard and batch watch is exactly batch wait.

Only a completed job exits 0. A job that ended failed or cancelled is printed too, but exits 4, with batch <id> failed (N of M items finished) and the job's error on stderr; partial results stay readable with batch results. This holds for batch wait, batch submit --wait and batch watch. Before 0.1.4, wait exited 0 for them: a script that relied on that checks .status in the printed job instead.

Inspect jobs

spicrawl batch get 01J9Z6V0Q8M4K2T7R3N5B1C9XA --json | jq '{status, progress}'
spicrawl batch list --status running
spicrawl batch list --all --json | jq -r '.batches[] | select(.status=="failed") | .id'

batch list prints {"batches": [...], "next_cursor": "..."} in JSON mode, and an ID STATUS NAME DONE SUBMITTED table otherwise. next_cursor is present only when more pages exist and you did not pass --all. With no jobs, human mode on a terminal prints no batch jobs on stderr and nothing on stdout (-q silences it); piped, or with --json, stdout is {"batches": []}.

Download results

spicrawl batch results <id> [--status S] [--all] [--limit N] [--cursor C] [-o FILE]

Fetches GET /v1/batch/{id}/results: one JSON object per finished item, in seq order. In JSON mode, or with -o, the lines are written unchanged as JSONL; in human mode a SEQ STATUS HTTP URL BYTES table is printed.

Each line has seq, external_id, url, status, attempts, http_status, request_id, bytes, duration_ms, result and error. result.content is the page as plain text in the item's format (html, markdown or text), so read it as it is. For --format json it is the JSON document as a string: pipe it through fromjson to get the object.

# Everything, to a file
spicrawl batch results "$id" --all -o results.jsonl

# Only failures, with their error codes
spicrawl batch results "$id" --status failed --all | jq -r '[.seq, .url, .error.code] | @tsv'

# The document of every succeeded item
spicrawl batch results "$id" --status succeeded --all | jq -r '.result.content'

Bodies are inlined only once the job is terminal. Results expire at the job's results_expire_at (72 hours after submission by default); after that the API answers ERR::REQUEST::BEYOND_RETENTION (HTTP 410, exit 4).

Change a running job

spicrawl batch retry "$id" && spicrawl batch wait "$id" --max-wait 15m
spicrawl batch cancel "$id" && spicrawl batch wait "$id" --max-wait 2m

Open jobs for crawls

id=$(spicrawl batch submit seeds.txt --open --name crawl | jq -r .id)
echo https://example.com/discovered/17 | spicrawl batch append "$id" -
spicrawl batch append "$id" more.jsonl
spicrawl batch close "$id" && spicrawl batch wait "$id"

Errors

See the batch guide for the job lifecycle and webhooks.