Logs, usage and status

Three read-only command groups tell you what happened, what it cost, and whether the API is reachable from this machine.

spicrawl logs --errors --limit 20      # recent failures
spicrawl logs get 01M0FK1EP2E7FBMWAR7W10SMXW
spicrawl usage summary                  # credits used against the plan
spicrawl status                         # API reachable, workers ready, key accepted

spicrawl logs

Lists recent API requests in the key's project, newest first (GET /v1/requests). With --all-projects, every project in the key's organization instead, which needs a key with the read scope. Records are kept for the organization's retention window (24 hours by default); older ones are gone.

spicrawl logs
spicrawl logs --errors --limit 20
spicrawl logs --all-projects --errors
spicrawl logs --status blocked --all --jsonl | jq -r .url

Output:

• Human mode: a TIME STATUS HTTP ENGINE CREDITS TOTAL_MS URL table, with a PROJECT column after TIME under --all-projects. When more pages exist and you did not pass --all, stderr says more records exist; pass --all to fetch every page.
• JSON mode: {"data": [...], "page": {...}}. page carries retention_hours, retained_since, has_more, next_before and next_before_id.
• --jsonl: one RequestLogEntry per line, no wrapper.

Each record has id, created_at, engine, status, http_status, error_code, error_detail, attempts, url, proxy, spans, blocked, credits_micro and features. Credits are in micro-credits (1 credit = 1,000,000).

# Error codes of the last day's failures, most frequent first
spicrawl logs --errors --all --jsonl | jq -r .error_code | sort | uniq -c | sort -rn

spicrawl logs get

Shows one request: outcome, error, latency spans, proxy and bot-block details (GET /v1/requests/{id}). The id is the X-Request-Id response header, the request_id in a scrape's JSON output, or the request_id of a problem document. Every failed command prints it in human mode as request id: <id> (spicrawl logs get <id>).

spicrawl logs get 01M0FK1EP2E7FBMWAR7W10SMXW
spicrawl logs get 01M0FK1EP2E7FBMWAR7W10SMXW --json | jq .spans

Spans are measured by different processes and do not sum to total_ms. A span shown as — was not measured, which is not the same as 0. An id older than the retention window returns ERR::REQUEST::NOT_FOUND or ERR::REQUEST::BEYOND_RETENTION (exit 4).

spicrawl usage

Billed usage for the key's organization, broken down by one dimension (GET /v1/usage). Needs a key with the read scope; otherwise ERR::AUTH::INSUFFICIENT_SCOPE (exit 3).

spicrawl usage
spicrawl usage --from 2026-08-01 --to 2026-09-01 --group-by engine
spicrawl usage --group-by project --metrics credits,requests
spicrawl usage --group-by key --metrics requests,client_cli

JSON mode prints the API's UsageBreakdown unchanged: period, group_by, groups, totals, unattributed, usage_as_of, stale_seconds, warnings. totals is the billed figure; trust it over the sum of groups. Credits are in micro-credits in JSON; the human table shows whole credits. credits is total spend, batch items included, in every grouping; with --group-by feature, the batch share of it is under batch.

With --group-by key the body is instead {period, group_by: "key", keys[]}. Each key has key_id, name, prefix, project_id, revoked, days[] ({day, metrics}) and window totals, read from the per-key rollup. Requests made without a key (the dashboard) are not in it, and a project-scoped key only sees its own project's keys. No caller IPs or countries are returned. The human table shows one row per key with window totals.

Besides the billing metrics, --metrics accepts requests_failed, requests_timeout, the per-feature counters (feature_screenshot, feature_pdf, ...) and the per-client counters (client_cli, client_mcp, client_sdk, client_other).

spicrawl usage summary

Current-period usage against the plan's credit allowance (GET /v1/usage/summary). When no subscription covers today, the organization is on the free plan (plan.code free, subscription_status none) and the period is its allowance month, which runs from the day of the month the organization was created. That is the window the monthly allowance is enforced over, not a billing period, so it is not a bill preview.

spicrawl usage summary
spicrawl usage summary --json | jq .allowance.remaining_micro

allowance is the monthly allowance as the API enforces it: limit_micro, limit_source (default or override), used_micro, remaining_micro, window_start and resets_at. Its used_micro includes credits held for work in flight (open browser sessions, unfinished batch items), so allowance.remaining_micro is what you can still spend.

credits is metered usage for the period: included_micro, used_micro, remaining_micro, overage_micro and used_basis_points. It includes batch spend, but not credits held for work still running.

spicrawl usage reconciliation

Compares billed usage for one UTC day with usage re-derived from the request log (GET /v1/usage/reconciliation). Report only; nothing is repaired.

spicrawl usage reconciliation
spicrawl usage reconciliation --day 2026-09-21 --json | jq .drift

Read interpretation before drift: when the request log holds no rows for the day, every billed row shows as drift, which is not evidence of overbilling. Batch usage is not reconciled here, because batch items are not in the request log: batch_items never appears, and a credits row compares only the non-batch part of the day's credits.

spicrawl status

Checks the whole path from this machine to a working API, in three calls:

spicrawl status
spicrawl status --json | jq .workers
spicrawl status --base-url http://localhost:8080 --api-key "$SPICRAWL_API_KEY"

{
  "api_reachable": true,
  "ready": true,
  "checks": { "postgres": "ok", "redis": "ok" },
  "cdp": "disabled",
  "key_valid": true,
  "key_source": "env",
  "base_url": "https://api.spicrawl.com",
  "workers": {
    "browser": { "present": 2, "ready": 2 },
    "egress": { "present": 1, "ready": 1 }
  }
}

cdp says whether GET /v1/browser (remote browser Coming soon) can serve a session on this deployment: enabled, disabled (the operator did not install the CDP browser; /v1/browser answers 501 or does not exist, and every other endpoint still works) or unknown (the browser worker could not be asked). Human output prints it as a cdp: line. key_valid is true, false, or null when the check could not tell (for example a network error mid-check). key_error explains a rejected or missing key; key is valid but lacks the read scope still counts as valid. workers is read only once the key is accepted; workers_error is set when that call failed.