spicrawl scrape

spicrawl scrape <url> calls POST /v1/scrape and prints the result. On a terminal it prints the document itself; piped, it prints one JSON object.

spicrawl scrape https://example.com/blog/launch --format markdown

Only the flags you set are sent. Everything else takes the API's default, or your project's stored default. Each flag below lists the API field it sets; see POST /v1/scrape for the full field rules.

Examples

# A page as clean markdown, for an LLM context window
spicrawl scrape https://example.com/blog/launch --format markdown

# A JS-rendered page, waiting for the element that holds the data
spicrawl scrape https://example.com/pricing --render --wait-for '.plans' --format markdown

# Through your own proxy, with the metadata on stderr
spicrawl scrape https://example.de/produkt/42 --proxy "$MY_PROXY_URL" --meta

# Selector extraction; prints only the extracted data
spicrawl scrape https://example.com/products/1 --extract '{"title":"h1","price":".price"}'

# Render JavaScript, retrying transient failures
spicrawl scrape https://news.example.com --render --retry 3

# Many URLs from a file, 8 at a time, one JSON line each; keep the failures
spicrawl scrape - --format markdown --concurrency 8 < urls.txt | jq -c 'select(.error)'

Flags

Target

Engine

Proxy

Two more options are on the way: stealth mode Coming soon (the Camoufox browser), and the managed proxy flags --premium-proxy, --country and --sticky-key Coming soon.

Waiting and page control

These need a browser engine (--render, a browser --engine, or --mode auto); otherwise the API returns ERR::REQUEST::INCOMPATIBLE_FLAGS.

Extraction

Each of these makes the API return its JSON envelope instead of the raw document.

Cleaning

Screenshots

Screenshots need chromium (--engine chromium); otherwise ERR::REQUEST::CAPABILITY_UNSUPPORTED.

Cache and cost

Output and retries (CLI only)

Output

Human mode (terminal)

Prints the document itself: markdown, html or text. With --extract, --ai, --ai-schema or --autoparse it prints only the extracted data, indented. API warnings go to stderr as warning: ....

--format pdf is binary and is never written to a terminal: pass -o FILE, or --json to get it base64-encoded. Without either, the command exits 2.

spicrawl scrape https://example.com/pricing --format markdown --meta

engine: fetch
credits: 1 (request cost 1)
cache: miss
target status: 200
proxy: direct
request id: 01J9ZQ4M7R3T8VX2K5N6P0B1CD

JSON mode (piped, or --json)

When the API answers with its JSON envelope (--format json, or any of --extract, --ai, --ai-schema, --autoparse, --links, --network-capture, --screenshot, or --actions containing a screenshot step or an evaluate with return_value), the envelope is printed as-is: url, final_url, status, content, credits, engine, proxy_source, warnings, data, links, screenshots, network.

When the API answers with the raw document (html, markdown, text), the CLI wraps it with the response headers:

{
  "url": "https://example.com/blog/launch",
  "final_url": "https://example.com/blog/launch",
  "status": 200,
  "engine": "fetch",
  "proxy_source": "direct",
  "credits_charged": 1,
  "request_cost": 1,
  "cache_state": "miss",
  "request_id": "01J9ZQ4M7R3T8VX2K5N6P0B1CD",
  "warnings": [],
  "content_type": "text/markdown; charset=utf-8",
  "content": "# Launch\n\nToday we ..."
}

status is the target site's status (X-Target-Status), not the API's. Binary documents (--format pdf) carry content_base64 instead of content.

spicrawl scrape https://example.com/blog/launch --format markdown | jq -r .content

Writing to a file with -o

-o FILE writes the document (the extracted data when extraction is on, raw bytes for PDF) to FILE and prints wrote N bytes to FILE on stderr. In JSON mode stdout still carries the metadata object, without the written field and with "output": "FILE" added.

spicrawl scrape https://example.com/report --engine chromium --format pdf -o report.pdf

Screenshots in the envelope are saved next to FILE as <FILE stem>.<label>.<format>, or into --screenshot-dir. Their base64 data is replaced by "file": "<path>" in the printed JSON. With neither -o nor --screenshot-dir, human mode leaves screenshots unsaved and says so on stderr; JSON mode keeps them base64 in the output.

spicrawl scrape https://example.com --engine chromium --screenshot --screenshot-full-page -o home.html
# writes home.html and home.<label>.png

Many URLs from stdin

spicrawl scrape - reads URLs from stdin, one per line. Blank lines and lines starting with # are skipped. It runs --concurrency requests at once (default 4) with the same flags for every URL, and prints one compact JSON line per URL in completion order, in every mode.

spicrawl scrape - --format markdown --concurrency 8 < urls.txt > pages.jsonl

Each success line is the result object above plus index, the 1-based position of the URL in the input. Each failure line is:

{"index":3,"url":"https://example.com/products/43","error":{"type":"https://docs.spicrawl.com/errors#UPSTREAM_CHALLENGE","title":"Target served a bot challenge","status":502,"code":"ERR::UPSTREAM::CHALLENGE","retryable":true,"doc_url":"https://docs.spicrawl.com/errors#UPSTREAM_CHALLENGE","target_status":403,"request_id":"01J9ZQ5B2C7D9EXAMPLE00000"}}

error is the API's problem document, or {"detail": "...", "exit_code": N} for a network or local failure.

• Exit code is 0 when every URL succeeded, else the exit code of the first failure. In human mode a N ok, M failed summary goes to stderr.
• -o is refused with - (exit 2); use --screenshot-dir for screenshots, which are saved as <index>.<label>.<format>.
• Empty stdin exits 2 with no URLs on stdin.

Errors

A failed scrape writes the API's problem document to stderr (JSON mode) or error:, hint:, target status:, retryable: and request id: lines (human mode), and exits with the code for its code. See exit codes.