Logs, usage and status

Three read-only command groups tell you what happened, what it cost, and whether the API is reachable from this machine.

spicrawl logs --errors --limit 20      # recent failures
spicrawl logs get 01M0FK1EP2E7FBMWAR7W10SMXW
spicrawl logs --watch                  # live request log (0.1.4 or later)
spicrawl usage summary                  # credits used against the plan
spicrawl status                         # API reachable, workers ready, key accepted, agents connected

spicrawl logs

Lists recent API requests in the key's project, newest first (GET /v1/requests). With --all-projects, every project in the key's organization instead, which needs a key with the read scope. Records are kept for the organization's retention window (24 hours by default); older ones are gone.

spicrawl logs
spicrawl logs --errors --limit 20
spicrawl logs --all-projects --errors
spicrawl logs --status blocked --all --jsonl | jq -r .url

Output:

• Human mode: a TIME STATUS HTTP ENGINE CREDITS TOTAL_MS URL table, with a PROJECT column after TIME under --all-projects. When more pages exist and you did not pass --all, stderr says more records exist; pass --all to fetch every page.
• JSON mode: {"data": [...], "page": {...}}. page carries retention_hours, retained_since, has_more, next_before and next_before_id.
• No matching requests: human mode on a terminal prints no requests in the last N hours on stderr and nothing on stdout (-q silences it). Piped, or with --json, stdout is {"data": [], "page": {...}}.
• --jsonl: one RequestLogEntry per line, no wrapper.

Each record has id, created_at, engine (the rendering tier: fetch, js or browser), status, http_status, error_code, error_detail, attempts, url, proxy, spans, blocked, credits_micro and features. Credits are in micro-credits (1 credit = 1,000,000).

# Error codes of the last day's failures, most frequent first
spicrawl logs --errors --all --jsonl | jq -r .error_code | sort | uniq -c | sort -rn

Live view: --watch

spicrawl logs --watch
spicrawl logs --watch --errors --all-projects
spicrawl logs --watch --json | jq -r 'select(.status != "success") | .url'

logs --watch (0.1.4 or later) polls the request log every --poll (default 2s) and shows new requests as they arrive. --errors, --status, --limit and --all-projects work as in logs; with --watch the default --limit is 200.

On a terminal it is a full-screen, scrollable view, newest request on top, and the selection stays on its row as new ones arrive. Keys:

The filters apply to the rows loaded so far (the newest --limit and everything since). To narrow the query itself, start with --errors or --status. A failed poll does not close the view: the error shows on the status bar and the poll is retried with backoff. A wrong key fails at once with its usual exit code, before the screen opens. NO_COLOR turns colour off and the selected row is marked with >.

The full-screen view needs a terminal on stdin and stdout. When either is not one, or with --json, -q, CI or TERM=dumb, logs --watch streams new requests as plain lines instead, oldest first: one JSON record per line with --json, which is the default when stdout is piped.

spicrawl logs get

Shows one request: outcome, error, latency spans, proxy and bot-block details (GET /v1/requests/{id}). The id is the X-Request-Id response header, the request_id in a scrape's JSON output, or the request_id of a problem document. Every failed command prints it in human mode as request id: <id> (spicrawl logs get <id>).

spicrawl logs get 01M0FK1EP2E7FBMWAR7W10SMXW
spicrawl logs get 01M0FK1EP2E7FBMWAR7W10SMXW --json | jq .spans

Spans are measured by different processes and do not sum to total_ms. A span shown as — was not measured, which is not the same as 0. An id older than the retention window returns ERR::REQUEST::NOT_FOUND or ERR::REQUEST::BEYOND_RETENTION (exit 4).

spicrawl usage

Billed usage for the key's organization, broken down by one dimension (GET /v1/usage). Needs a key with the read scope; otherwise ERR::AUTH::INSUFFICIENT_SCOPE (exit 3).

spicrawl usage
spicrawl usage --from 2026-08-01 --to 2026-09-01 --group-by engine
spicrawl usage --group-by project --metrics credits,requests
spicrawl usage --group-by key --metrics requests,client_cli

JSON mode prints the API's UsageBreakdown unchanged: period, group_by, groups, totals, unattributed, usage_as_of, stale_seconds, warnings. totals is the billed figure; trust it over the sum of groups. Credits are in micro-credits in JSON; the human table shows whole credits. credits is total spend, batch items included, in every grouping; with --group-by feature, the batch share of it is under batch.

With --group-by key the body is instead {period, group_by: "key", keys[]}. Each key has key_id, name, prefix, project_id, revoked, days[] ({day, metrics}) and window totals, read from the per-key rollup. Requests made without a key (the dashboard) are not in it, and a project-scoped key only sees its own project's keys. No caller IPs or countries are returned. The human table shows one row per key with window totals.

Besides the billing metrics, --metrics accepts requests_failed, requests_timeout, the per-feature counters (feature_screenshot, feature_pdf, ...) and the per-client counters (client_cli, client_mcp, client_sdk, client_other).

spicrawl usage summary

Current-period usage against the plan's credit allowance (GET /v1/usage/summary). When no subscription covers today, the organization is on the free plan (plan.code free, subscription_status none) and the period is its allowance month, which runs from the day of the month the organization was created. That is the window the monthly allowance is enforced over, not a billing period, so it is not a bill preview.

spicrawl usage summary
spicrawl usage summary --json | jq .allowance.remaining_micro

allowance is the monthly allowance as the API enforces it: limit_micro, limit_source (default or override), used_micro, remaining_micro, window_start and resets_at. Its used_micro includes credits held for work in flight (open browser sessions, unfinished batch items), so allowance.remaining_micro is what you can still spend. It includes extra_micro, extra credits granted on top of the monthly limit.

credits is metered usage for the period: included_micro, used_micro, remaining_micro, overage_micro and used_basis_points. It includes batch spend, but not credits held for work still running.

spicrawl usage reconciliation

Compares billed usage for one UTC day with usage re-derived from the request log (GET /v1/usage/reconciliation). Report only; nothing is repaired.

spicrawl usage reconciliation
spicrawl usage reconciliation --day 2026-09-21 --json | jq .drift

Read interpretation before drift: when the request log holds no rows for the day, every billed row shows as drift, which is not evidence of overbilling. Batch usage is not reconciled here, because batch items are not in the request log: batch_items never appears, and a credits row compares only the non-batch part of the day's credits.

spicrawl status

Checks the whole path from this machine to a working API, in three calls and one look at the agent config files:

spicrawl status
spicrawl status --json | jq .workers
spicrawl status --base-url http://localhost:8080 --api-key "$SPICRAWL_API_KEY"

{
  "api_reachable": true,
  "ready": true,
  "checks": { "postgres": "ok", "redis": "ok" },
  "cdp": "disabled",
  "key_valid": true,
  "key_source": "env",
  "base_url": "https://api.spicrawl.com",
  "workers": {
    "browser": { "present": 2, "ready": 2 },
    "egress": { "present": 1, "ready": 1 }
  },
  "mcp": { "detected": ["claude", "cursor"], "connected": ["claude", "cursor"], "needs_env": [], "stale": [] },
  "repairs": []
}

cdp says whether GET /v1/browser (remote browser) can serve a session on this deployment: enabled, disabled (the operator did not install the CDP browser; /v1/browser answers 501 or does not exist, and every other endpoint still works) or unknown (the browser worker could not be asked). Human output prints it as a cdp: line. key_valid is true, false, or null when the check could not tell (for example a network error mid-check). key_error explains a rejected or missing key; key is valid but lacks the read scope still counts as valid. workers is read only once the key is accepted; workers_error is set when that call failed.

mcp lists agent client ids (claude, cursor, vscode, codex):

A client in needs_env or stale is not connected, and it fails the mcp check. A client in none of the three has no Spicrawl entry. Human output adds an mcp: line, for example connected: Claude Code, not configured: Cursor, needs $SPICRAWL_API_KEY, which is not set here: Claude Code or has a different key than the one in use: Claude Code.

A connection that fails during the key check exits with the transport error's code (12 if it timed out or dropped). The report is on stdout whatever the exit code. Behaviour change in 0.1.4: before it, a reachable API that reported not ready exited 0, and status had no mcp or repairs. A script that needs the old result reads ready and checks from --json instead of the exit code.

Repairs

For every failed check, status prints a repair command, and --json has them in the repairs array of {check, command, unattended_safe, why}, in the order to run them. The array is always present and is [] when every check passed. On a terminal, status draws a checklist with the repairs under it. With NO_COLOR, TERM=dumb or CI set, the entries print as a repairs: block of check:, command:, unattended_safe: and why: lines. Piped, as an agent runs it, status prints the JSON.

"repairs": [
  {
    "check": "key",
    "command": "spicrawl login --api-key <YOUR_API_KEY>",
    "unattended_safe": false,
    "why": "no API key is configured: create one at https://app.spicrawl.com/dashboard/keys and supply it"
  },
  {
    "check": "mcp",
    "command": "spicrawl init --all --yes",
    "unattended_safe": true,
    "why": "no Spicrawl MCP server is configured for Claude Code; this writes the MCP entry and the skill for every client found here"
  }
]

unattended_safe: true means an agent may run the command without asking. false means the user must act (supply a key) or decide, so show them the command instead. A command holds the placeholder <YOUR_API_KEY>, never a key. When the CLI was started by npx, commands name npx -y @spicrawl/cli@latest instead of spicrawl.

A needs_env or stale repair rewrites the entry with the key saved in the CLI's config file (by spicrawl login or init --api-key) and keeps the old file as <file>.bak, so nothing needs exporting and --force is not needed. A key that is only in $SPICRAWL_API_KEY is not saved, so init keeps writing ${SPICRAWL_API_KEY} references for it: see Agent setup.