Scripting with the CLI

Recipes for using spicrawl in shell scripts, agent tool calls and CI: jq pipelines, URL files, exit-code branching, retries and per-URL files.

Every recipe here relies on three rules: stdout is data only, output is JSON whenever stdout is not a terminal, and the exit code says what went wrong. Inside $(...) or a pipe you always get JSON, so jq always works. In that mode stderr is empty on success and holds only the failure document, so 2>err.json is always safe to parse (0.1.4 or later).

spicrawl scrape https://example.com/blog/launch --format markdown | jq -r .content

Pipe to jq

# Just the markdown
spicrawl scrape https://example.com/blog/launch --format markdown | jq -r .content

# Credits charged and whether the cache served it
spicrawl scrape https://example.com/pricing --render | jq '{credits_charged, cache_state}'

# Extracted fields only
spicrawl scrape https://example.com/products/42 \
  --extract '{"title":"h1","price":".price"}' | jq .data

# All links on a page, one per line
spicrawl scrape https://example.com --links | jq -r '.links[]'

With --extract, --ai, --autoparse, --links, --screenshot, --network-capture or --format json, the JSON is the API's envelope (fields content, data, links, credits, ...). Otherwise it is the CLI's wrapper (fields content, status, credits_charged, request_id, ...). See scrape output.

Loop over a file of URLs

Let the CLI do the loop: spicrawl scrape - reads one URL per line, runs them in parallel, and prints one JSON line per URL.

spicrawl scrape - --format markdown --concurrency 8 < urls.txt > pages.jsonl

# Which failed, and why (a page that answered 404 has target_error, not error)
jq -r 'select(.error or .target_error) | [.index, .url, (.error.code // .target_error)] | @tsv' pages.jsonl

# Retry only the failures, rendered in a browser
jq -r 'select(.error or .target_error) | .url' pages.jsonl \
  | spicrawl scrape - --format markdown --render > retried.jsonl

Lines arrive in completion order; sort by .index to restore input order: jq -s -c 'sort_by(.index)[]' pages.jsonl. The exit code is 0 only if every URL succeeded.

For thousands of URLs that should keep running after your shell exits, use spicrawl batch instead. Batch returns html, markdown or text (--format), but not extraction or screenshots.

Save markdown per URL

mkdir -p pages
spicrawl scrape - --format markdown --concurrency 8 < urls.txt \
  | jq -c 'select((.error or .target_error) | not)' \
  | while IFS= read -r line; do
      name=$(printf '%s' "$line" | jq -r '.url | sub("^https?://"; "") | gsub("[^A-Za-z0-9._-]"; "_")')
      printf '%s' "$line" | jq -r .content > "pages/$name.md"
    done

For a handful of URLs, a plain loop with -o is clearer:

while IFS= read -r url; do
  name=$(printf '%s' "$url" | sed -E 's#^https?://##; s#[^A-Za-z0-9._-]#_#g')
  spicrawl scrape "$url" --format markdown -o "pages/$name.md" -q || echo "failed: $url" >&2
done < urls.txt

Branch on exit codes

spicrawl scrape "$url" --format markdown --json -o page.md >/dev/null 2>err.json
case $? in
  0)  ;;                                            # page.md written
  2)  echo "bad flags" >&2; exit 2 ;;
  3)  echo "set SPICRAWL_API_KEY" >&2; exit 3 ;;
  4)  jq -r '.detail // .error' err.json >&2 ;;     # fix the request; do not retry
  5)  sleep "$(jq -r '.retry_after_seconds // 30' err.json)" ;;
  6)  spicrawl scrape "$url" --format markdown --render -o page.md ;;
  7|8) echo "retryable: $(jq -r .retryable err.json)" >&2 ;;
  10) echo "cannot reach the API; nothing was sent" >&2 ;;
  12) echo "timed out or dropped: may have been billed, check spicrawl logs" >&2 ;;
  130) exit 130 ;;                                  # Ctrl-C
  *)  jq -r '.code // .error' err.json >&2 ;;
esac

--json keeps err.json machine-readable when you run the script from a terminal; without a terminal it is JSON anyway. A page that answers 404 or 503 exits 6 too, with the document still on stdout: add --allowed-status 404 when that is a normal answer. On exit 12 the request may have run, so look at spicrawl logs before you retry.

Retry wrapper

For transient failures, prefer the built-in --retry N: it retries only errors the API marks retryable, and waits retry_after_seconds when the API gives one (otherwise 1 s, doubling up to 30 s).

spicrawl scrape https://example.com/pricing --render --format markdown --retry 4

To escalate between attempts (a plain fetch first, a render on a bot challenge), write the loop yourself:

scrape_with_escalation() {
  url=$1
  for flags in "" "--render" "--render --proxy $MY_PROXY_URL"; do
    # shellcheck disable=SC2086
    out=$(spicrawl scrape "$url" --format markdown $flags 2>err.json)
    code=$?
    if [ "$code" -eq 0 ]; then
      printf '%s\n' "$out" | jq -r .content
      return 0
    fi
    case $code in
      6|7|8) continue ;;          # site, proxy or rendering: try the next tier
      5) sleep "$(jq -r '.retry_after_seconds // 10' err.json)" ;;
      *) jq -r '.code // .error' err.json >&2; return "$code" ;;
    esac
  done
  return 6
}

scrape_with_escalation https://example.com/products/42 > product.md

--mode auto climbs for you server-side, from a plain fetch to stronger rendering, and bills only the step that succeeded.

Agent tool calls

An agent that drives a shell should:

Read the key from the environment. Do not run spicrawl login inside the agent.
Read stderr only when the exit code is non-zero: when stdout is not a terminal it is empty on success. -q also silences status lines and the spinner if the agent runs the command in a terminal.
Cap spend with --max-cost N: a request that could cost more fails with ERR::LIMIT::MAX_COST_EXCEEDED (exit 5) at 0 credits.
Pass --yes to destructive commands; without a terminal they refuse with exit 2.
Validate hand-built bodies against spicrawl schema scrape and send them with --body @file.

spicrawl scrape https://example.com/products/42 --format markdown --max-cost 5 -q

CI

Pass the key as a secret environment variable. Nothing is written to disk and nothing prompts. With CI set the CLI draws no spinner, colour or summary line, so the log holds only plain lines.

.github/workflows/snapshot.yml

name: Nightly pricing snapshot
on:
  schedule:
    - cron: "0 3 * * *"
jobs:
  snapshot:
    runs-on: ubuntu-latest
    env:
      SPICRAWL_API_KEY: ${{ secrets.SPICRAWL_API_KEY }}
    steps:
      - uses: actions/checkout@v4
      - name: Install spicrawl
        run: npm install -g @spicrawl/cli
      - name: Check the key
        run: spicrawl auth status
      - name: Scrape
        run: |
          spicrawl scrape - --format markdown --concurrency 4 < urls.txt > pages.jsonl \
            || { jq -c 'select(.error or .target_error) | {url, error: (.error.code // .target_error)}' pages.jsonl; exit 1; }
      - uses: actions/upload-artifact@v4
        with:
          name: pages
          path: pages.jsonl

spicrawl auth status exits 3 on a missing or revoked key, so the job fails before any scraping. spicrawl scrape - exits non-zero if any URL failed; the jq fallback prints each failed URL and its error code to the log before failing the step.