spicrawl scrape

spicrawl scrape <url> calls POST /v1/scrape and prints the result. On a terminal it prints the document itself; piped, it prints one JSON object.

spicrawl scrape https://example.com/blog/launch --format markdown

Only the flags you set are sent. Everything else takes the API's default, or your project's stored default. Each flag below lists the API field it sets; see POST /v1/scrape for the full field rules.

Examples

# A page as clean markdown, for an LLM context window
spicrawl scrape https://example.com/blog/launch --format markdown

# A JS-rendered page, waiting for the element that holds the data
spicrawl scrape https://example.com/pricing --render --wait-for '.plans' --format markdown

# Through your own proxy, with the metadata on stderr
spicrawl scrape https://example.de/produkt/42 --proxy "$MY_PROXY_URL" --meta

# A Cloudflare-protected page, through your own (residential) proxy
spicrawl scrape https://example.com/products/42 --proxy "$MY_PROXY_URL" --format markdown

# Selector extraction; prints only the extracted data
spicrawl scrape https://example.com/products/1 --extract '{"title":"h1","price":".price"}'

# Render JavaScript, retrying transient failures
spicrawl scrape https://news.example.com --render --retry 3

# Many URLs from a file, 8 at a time, one JSON line each; keep the failures
spicrawl scrape - --format markdown --concurrency 8 < urls.txt | jq -c 'select(.error or .target_error)'

Flags

Target

Rendering

You do not pick a rendering tier. The request picks it, and the result names the one that ran: fetch (a plain fetch, no JavaScript), js (JavaScript rendering) or browser (full-browser rendering). It is the engine field in JSON and the engine: line of --meta. See response headers.

Proxy

Two more options are on the way: stealth mode Coming soon (full-browser rendering with a hardened fingerprint), and the managed proxy flags --premium-proxy, --country and --sticky-key Coming soon.

Cloudflare-protected pages, and pages behind similar managed challenges, are handled automatically when you bring your own --proxy (residential works best): a challenged page is retried on stronger rendering on the same exit, and you get the real page. This is not guaranteed on every site. Automatic handling on managed --premium-proxy exits is Coming soon. On the default shared pool a challenged page can return ERR::UPSTREAM::CHALLENGE; retry with your own residential --proxy. If a request uses your own proxy and hits a blocking challenge that Spicrawl solves, your proxy's address and credentials are shared with the third-party solving service for that solve.

Waiting and page control

These need a browser (--render or --mode auto); otherwise the API returns ERR::REQUEST::INCOMPATIBLE_FLAGS.

Extraction

Each of these makes the API return its JSON envelope instead of the raw document.

Cleaning

Screenshots

A screenshot is rendered by a full browser, chosen for you. If it cannot be taken, the API returns ERR::REQUEST::CAPABILITY_UNSUPPORTED at 0 credits.

Cache and cost

Output and retries (CLI only)

Output

Human mode (terminal)

Prints the document itself: markdown, html or text. With --extract, --ai, --ai-schema or --autoparse it prints only the extracted data, indented, with the fields in the order the API returned them (0.1.4 or later; earlier versions sorted them alphabetically). With --links or --network-capture it prints that data after the document, under --- links (N) --- and --- network (N) --- lines. API warnings go to stderr as warning: ...: they print even with -q, and never in JSON mode, where they are in the output's warnings field.

--format pdf is binary and is never written to a terminal: pass -o FILE, or --json to get it base64-encoded. Without either, the command exits 2.

spicrawl scrape https://example.com/pricing --format markdown --meta

engine: fetch
credits: 1 (request cost 1)
cache: miss
target status: 200
proxy: direct
final url: https://example.com/pricing
request id: 01J9ZQ4M7R3T8VX2K5N6P0B1CD

engine: is a rendering tier (fetch, js or browser) whatever the API sent. --json and --jsonl print the API's values unchanged.

JSON mode (piped, or --json)

When the API answers with its JSON envelope (--format json, or any of --extract, --ai, --ai-schema, --autoparse, --links, --network-capture, --screenshot, or --actions containing a screenshot step or an evaluate with return_value), the envelope is printed as-is, in the API's key order, and large integers are not rounded (0.1.4 or later): url, final_url, status, content, credits, engine, proxy_source, warnings, data, links, screenshots, network.

When the API answers with the raw document (html, markdown, text), the CLI wraps it with the response headers:

{
  "cache_state": "miss",
  "content": "# Launch\n\nToday we ...",
  "content_type": "text/markdown; charset=utf-8",
  "credits": 1,
  "credits_charged": 1,
  "engine": "fetch",
  "final_url": "https://example.com/blog/launch",
  "proxy_source": "direct",
  "request_cost": 1,
  "request_id": "01J9ZQ4M7R3T8VX2K5N6P0B1CD",
  "status": 200,
  "url": "https://example.com/blog/launch",
  "warnings": []
}

status is the target site's status (X-Target-Status), not the API's. credits repeats credits_charged under the envelope's name. Binary documents (--format pdf) carry content_base64 instead of content. When the target's status is outside 2xx, the object also has target_error (see target status errors).

stderr in JSON mode is empty on success. On a failure it holds only the failure document, so spicrawl scrape ... 2>err.json; jq . err.json always parses: the API's problem document (code, retryable, request_id, ...), or {"error": "...", "exit_code": N} when the failure did not come from the API (a usage error, a network failure). Warnings, wrote N bytes and the deprecation notices are human-mode output only (0.1.4 or later; earlier versions printed them as prose on stderr, sometimes before the problem document).

spicrawl scrape https://example.com/blog/launch --format markdown | jq -r .content

Target status errors

A site that answers with an error is still a successful API call: the API returns HTTP 200 with the site's status in X-Target-Status (status in the output). The CLI treats a target status outside 2xx as a failure unless --allowed-status lists it. The document is still printed, stderr gets error: target returned 404 (pass --allowed-status 404 to accept it), and the exit code is 6. In JSON mode the printed object also has "target_error": "target returned 404".

spicrawl scrape https://example.com/old-page --allowed-status 404   # exits 0 on a 404 page

Writing to a file with -o

-o FILE writes the document (the extracted data when extraction is on, raw bytes for PDF) to FILE. In human mode it prints wrote N bytes to FILE on stderr (not with -q, never in JSON mode). The write is atomic: a temporary file in the same directory, then a rename, so a failure never truncates an existing FILE. -o - means stdout. In JSON mode stdout still carries the metadata object, without the written field and with "output": "FILE" added.

FILE, and the screenshots next to it, are written only when the target answered 2xx or its status is in --allowed-status (0.1.4 or later). On any other target status the earlier file is left alone, human mode prints warning: not writing FILE: the target failed (--allowed-status accepts it), and the command exits 6. If the write itself fails (no such directory, no permission), the billed result is printed to stdout as JSON instead of being lost, the error goes to stderr, and the command exits 1.

spicrawl scrape https://example.com/report --format pdf -o report.pdf

Screenshots in the envelope are saved next to FILE as <FILE stem>.<label>.<format>, or into --screenshot-dir. Their base64 data is replaced by "file": "<path>" in the printed JSON. With neither -o nor --screenshot-dir, human mode leaves screenshots unsaved and says so on stderr; JSON mode keeps them base64 in the output.

spicrawl scrape https://example.com --screenshot --screenshot-full-page -o home.html
# writes home.html and home.<label>.png

Live progress on a terminal

When stderr is a terminal (0.1.4 or later), a wait longer than about 300 ms shows a spinner there with a label and the elapsed time, for example ⠹ Scraping example.com/pricing · 4.2s. scrape and scrape - end with a one-line summary on stderr: ✓ or ✗, the page, the target status, the credits charged and the duration, for example ✓ example.com/pricing 200 · 1 credit · 4.2s. A request that finishes in under 300 ms shows the summary only. scrape - shows Scraping 2/3 done (1 failed) while it runs and ends with ✗ 3 URLs 2 ok · 1 failed · 1.5s. An API error is drawn as a short block that keeps the ERR:: code and the hint, with the target status, retryable, the request id (spicrawl logs get <id>) and a docs link.

stdout is never touched: the same bytes come out with or without it. It is off, and stderr is exactly what the sections above describe, when:

• stderr is not a terminal (a pipe, a file, a CI log or an agent's tool call),
• you pass --json or -q,
• NO_COLOR is set to anything non-empty, or CLICOLOR=0,
• TERM=dumb, or
• CI is set to anything but empty, 0 or false.

Piping stdout alone does not turn it off: spicrawl scrape ... | jq in a terminal still shows the spinner. Outside a UTF-8 locale the glyphs fall back to ASCII (+ x and a | / - \ spinner).

Many URLs from stdin

spicrawl scrape - reads URLs from stdin, one per line. Blank lines and lines starting with # are skipped. It runs --concurrency requests at once (default 4) with the same flags for every URL, and prints one compact JSON line per URL in completion order, in every mode.

spicrawl scrape - --format markdown --concurrency 8 < urls.txt > pages.jsonl

Each success line is the result object above plus index, the 1-based position of the URL in the input. Each failure line is:

{"index":3,"url":"https://example.com/products/43","error":{"type":"https://docs.spicrawl.com/errors#UPSTREAM_CHALLENGE","title":"Target served a bot challenge","status":502,"code":"ERR::UPSTREAM::CHALLENGE","retryable":true,"doc_url":"https://docs.spicrawl.com/errors#UPSTREAM_CHALLENGE","target_status":403,"request_id":"01J9ZQ5B2C7D9EXAMPLE00000"}}

error is the API's problem document, or {"detail": "...", "exit_code": N} for a network or local failure.

• Exit code is 0 when every URL succeeded, else the exit code of the first failure. A target status outside 2xx (and not in --allowed-status) is a failure: its line keeps the result and adds target_error, and the code is 6. In human mode a N ok, M failed line (on a live terminal, the summary line above) and error: M of N URLs failed go to stderr; in JSON mode stderr holds {"error": "M of N URLs failed", "exit_code": N}.
• If stdout cannot be written the run stops at once with exit 1 instead of billing the remaining URLs. Ctrl-C stops it too, including while it waits for stdin (exit 130).
• -o is refused with - (exit 2); use --screenshot-dir for screenshots, which are saved as <index>.<label>.<format>.
• Empty stdin exits 2 with no URLs on stdin.

Errors

A failed scrape exits with the code for its code. In JSON mode stderr holds the API's problem document, or {"error": "...", "exit_code": N} for a failure that did not come from the API. In human mode it holds error:, hint:, target status:, retryable: and request id: lines, which a terminal draws as the block described under live progress. See exit codes.