# Rate limits > How Spicrawl limits request rate, concurrency, live sessions and monthly credits, the headers that report each limit, and how to back off. Source: https://docs.spicrawl.com/rate-limits Spicrawl applies four independent limits. Three return `429` and are retryable after `Retry-After`; the monthly credit ceiling returns `402` and is not. Read the limit headers on every response and pace your client from them instead of hard-coding numbers. | Limit | Scope | Error | HTTP | Retryable | `Retry-After` | | ------------------- | ------------------------------------------------------------------ | ------------------------------------------------------------------------ | ---- | --------- | ----------------------------- | | Request rate | per API key | [`ERR::LIMIT::RATE_LIMITED`](https://docs.spicrawl.com/errors.md#LIMIT_RATE_LIMITED) | 429 | yes | seconds until a token is free | | Concurrent requests | per organization | [`ERR::LIMIT::CONCURRENCY_EXCEEDED`](https://docs.spicrawl.com/errors.md#LIMIT_CONCURRENCY_EXCEEDED) | 429 | yes | `1` on `/v1/scrape` | | Live sessions | per organization, max 100 | [`ERR::LIMIT::SESSIONS_EXCEEDED`](https://docs.spicrawl.com/errors.md#LIMIT_SESSIONS_EXCEEDED) | 429 | yes | `30` | | Monthly credits | per organization, per allowance month (org's creation anniversary) | [`ERR::LIMIT::QUOTA_EXCEEDED`](https://docs.spicrawl.com/errors.md#LIMIT_QUOTA_EXCEEDED) | 402 | no | none | Every limit refusal costs 0 credits. ## Headers ```text X-RateLimit-Limit: 10 X-RateLimit-Remaining: 7 Concurrency-Limit: 5 Concurrency-Remaining: 4 ``` | Header | Meaning | | ----------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `X-RateLimit-Limit` | Requests per second allowed for this key. | | `X-RateLimit-Remaining` | Tokens left in the bucket right now. | | `X-RateLimit-Reset` | Seconds until you may retry. Sent **only** when the limiter imposed a wait, so its absence means you were not limited. | | `Concurrency-Limit` | Concurrent in-flight requests allowed for your organization. | | `Concurrency-Remaining` | Concurrent slots left. `0` on a `CONCURRENCY_EXCEEDED` refusal. | | `Retry-After` | Seconds to wait before retrying. Present on retryable limit errors, and mirrored in the body as `retry_after_seconds`. | The values shown above are examples. Your plan, organization or key can set different limits, and the headers always report the ones that apply. ## Request rate The rate limit is a token bucket per API key. Each request takes one token; the bucket refills at `X-RateLimit-Limit` tokens per second and allows short bursts above that. A cache hit (`Cache-State: hit`) still takes a token. When the bucket is empty you get `429 ERR::LIMIT::RATE_LIMITED` with `Retry-After` and `X-RateLimit-Reset`. Wait that many seconds and retry. `ERR::PROXY::RATE_LIMITED` is different: it is `502`, and it means the proxy exit was rate limited by its provider, not that you exceeded your API limit. See [Errors](https://docs.spicrawl.com/errors.md#LIMIT_RATE_LIMITED). ## Concurrency Concurrency counts requests in flight at the same moment across every key in your organization. A browser render holds its slot for as long as it runs, so slow pages use up concurrency faster than rate. * `/v1/scrape`: when every slot is busy you get `429 ERR::LIMIT::CONCURRENCY_EXCEEDED`, `Retry-After: 1` and `Concurrency-Remaining: 0`. Cap your worker pool at `Concurrency-Limit`. * Batch jobs run under their own per-job `concurrency` (1 to 50) and do not need client-side pacing. ## Live sessions An organization may hold at most 100 live sessions created with `POST /v1/sessions`. The 101st is refused with `429 ERR::LIMIT::SESSIONS_EXCEEDED` and `Retry-After: 30`. Release sessions you are done with (`POST /v1/sessions/{sessionID}/release`) instead of waiting for their TTL. Sending two requests to the same session at once is a different error: `409 ERR::SESSION::BUSY`, retryable after `Retry-After`. ## Monthly credit ceiling Each organization has a monthly credit ceiling, counted per allowance month: the month runs from the day the org was created and resets at 00:00 UTC on that day every month (the last day of a shorter month for an org created on the 29th, 30th or 31st). Before a request runs, Spicrawl reserves its dearest possible cost (the top rung under `mode=auto`, every item's dearest outcome for a batch, or the whole `session_ttl` for a browser session). When the reservation does not fit, the request fails with `402 ERR::LIMIT::QUOTA_EXCEEDED`, naming the exact reset date. This is not retryable. Backing off does not help until the ceiling is raised or the allowance resets. Stop the job and alert a human. See [Credits](https://docs.spicrawl.com/credits.md). ## Backoff recipe 1. On a `429`, wait `Retry-After` seconds, plus up to one second of jitter. 2. On any other error with `retryable: true` and no `Retry-After`, back off exponentially: 1 s, 2 s, 4 s. 3. Stop after 3 retries, and never retry `retryable: false` or `402`. 4. Keep in-flight requests at or below `Concurrency-Limit`. ```python title="backoff.py" import os import random import time import httpx API = "https://api.spicrawl.com/v1/scrape" HEADERS = {"Authorization": f"Bearer {os.environ['SPICRAWL_API_KEY']}"} def scrape(client: httpx.Client, body: dict, max_retries: int = 3) -> httpx.Response: for attempt in range(max_retries + 1): r = client.post(API, json=body, headers=HEADERS, timeout=120) if r.is_success: return r problem = r.json() if r.status_code == 402 or not problem.get("retryable") or attempt == max_retries: raise RuntimeError(f"{problem['code']}: {problem.get('detail')}") retry_after = r.headers.get("Retry-After") wait = int(retry_after) if retry_after else 2**attempt time.sleep(wait + random.random()) raise AssertionError("unreachable") with httpx.Client() as client: page = scrape(client, {"url": "https://example.com/products/42", "response_format": "markdown"}) print(page.headers["X-Credits-Charged"], page.text[:200]) ``` ```typescript title="backoff.ts" const API = "https://api.spicrawl.com/v1/scrape"; async function scrape(body: object, maxRetries = 3): Promise { for (let attempt = 0; attempt <= maxRetries; attempt++) { const r = await fetch(API, { method: "POST", headers: { Authorization: `Bearer ${process.env.SPICRAWL_API_KEY}`, "Content-Type": "application/json", }, body: JSON.stringify(body), }); if (r.ok) return r; const problem = await r.json(); if (r.status === 402 || !problem.retryable || attempt === maxRetries) { throw new Error(`${problem.code}: ${problem.detail}`); } const retryAfter = Number(r.headers.get("Retry-After") ?? 0); const waitMs = (retryAfter || 2 ** attempt) * 1000 + Math.random() * 1000; await new Promise((resolve) => setTimeout(resolve, waitMs)); } throw new Error("unreachable"); } const page = await scrape({ url: "https://example.com/products/42", response_format: "markdown" }); console.log(page.headers.get("X-Credits-Charged"), (await page.text()).slice(0, 200)); ``` ```bash title="CLI" # --retry N is the total number of attempts (1 = no retry). The CLI retries only # retryable errors, honours Retry-After, and backs off exponentially up to 30 s. spicrawl scrape https://example.com/products/42 --format markdown --retry 4 ``` Retrying failed requests is free, because failures cost 0 credits. Retrying after a network timeout, when you did not see the response, can bill twice: `POST /v1/scrape` has no idempotency key.