Rate limits

Spicrawl applies four independent limits. Three return 429 and are retryable after Retry-After; the monthly credit ceiling returns 402 and is not. Read the limit headers on every response and pace your client from them instead of hard-coding numbers.

Every limit refusal costs 0 credits.

Headers

X-RateLimit-Limit: 10
X-RateLimit-Remaining: 7
Concurrency-Limit: 5
Concurrency-Remaining: 4

The values shown above are examples. Your plan, organization or key can set different limits, and the headers always report the ones that apply.

Request rate

The rate limit is a token bucket per API key. Each request takes one token; the bucket refills at X-RateLimit-Limit tokens per second and allows short bursts above that. A cache hit (Cache-State: hit) still takes a token.

When the bucket is empty you get 429 ERR::LIMIT::RATE_LIMITED with Retry-After and X-RateLimit-Reset. Wait that many seconds and retry.

ERR::PROXY::RATE_LIMITED is different: it is 502, and it means the proxy exit was rate limited by its provider, not that you exceeded your API limit. See Errors.

Concurrency

Concurrency counts requests in flight at the same moment across every key in your organization. A browser render holds its slot for as long as it runs, so slow pages use up concurrency faster than rate.

• /v1/scrape: when every slot is busy you get 429 ERR::LIMIT::CONCURRENCY_EXCEEDED, Retry-After: 1 and Concurrency-Remaining: 0. Cap your worker pool at Concurrency-Limit.
• Batch jobs run under their own per-job concurrency (1 to 50) and do not need client-side pacing.

Live sessions

An organization may hold at most 100 live sessions created with POST /v1/sessions. The 101st is refused with 429 ERR::LIMIT::SESSIONS_EXCEEDED and Retry-After: 30. Release sessions you are done with (POST /v1/sessions/{sessionID}/release) instead of waiting for their TTL.

Sending two requests to the same session at once is a different error: 409 ERR::SESSION::BUSY, retryable after Retry-After.

Monthly credit ceiling

Each organization has a monthly credit ceiling, counted per allowance month: the month runs from the day the org was created and resets at 00:00 UTC on that day every month (the last day of a shorter month for an org created on the 29th, 30th or 31st). Before a request runs, Spicrawl reserves its dearest possible cost (the top rung under mode=auto, every item's dearest outcome for a batch, or the whole session_ttl for a browser session). When the reservation does not fit, the request fails with 402 ERR::LIMIT::QUOTA_EXCEEDED, naming the exact reset date.

This is not retryable. Backing off does not help until the ceiling is raised or the allowance resets. Stop the job and alert a human. See Credits.

Backoff recipe

1. On a 429, wait Retry-After seconds, plus up to one second of jitter.
2. On any other error with retryable: true and no Retry-After, back off exponentially: 1 s, 2 s, 4 s.
3. Stop after 3 retries, and never retry retryable: false or 402.
4. Keep in-flight requests at or below Concurrency-Limit.

import os
import random
import time

import httpx

API = "https://api.spicrawl.com/v1/scrape"
HEADERS = {"Authorization": f"Bearer {os.environ['SPICRAWL_API_KEY']}"}


def scrape(client: httpx.Client, body: dict, max_retries: int = 3) -> httpx.Response:
    for attempt in range(max_retries + 1):
        r = client.post(API, json=body, headers=HEADERS, timeout=120)
        if r.is_success:
            return r
        problem = r.json()
        if r.status_code == 402 or not problem.get("retryable") or attempt == max_retries:
            raise RuntimeError(f"{problem['code']}: {problem.get('detail')}")
        retry_after = r.headers.get("Retry-After")
        wait = int(retry_after) if retry_after else 2**attempt
        time.sleep(wait + random.random())
    raise AssertionError("unreachable")


with httpx.Client() as client:
    page = scrape(client, {"url": "https://example.com/products/42", "response_format": "markdown"})
    print(page.headers["X-Credits-Charged"], page.text[:200])

Retrying failed requests is free, because failures cost 0 credits. Retrying after a network timeout, when you did not see the response, can bill twice: POST /v1/scrape has no idempotency key.