Click, type and scroll with browser actions

Use this when the content you need appears only after you interact with the page: a "Load more" button, a cookie wall, a tab, a search box, a login form, or an infinite-scroll feed. actions is an ordered list of steps the browser runs after the page loads and before it is captured. The whole list is validated before anything is spent.

Actions are browser-only. Set js_render: true or pin a browser engine; on the fetch tier they are a 400.

Minimal request

Click "Load more", wait for the new rows, and return the page as markdown:

curl https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com/products?category=lamps",
    "js_render": true,
    "response_format": "markdown",
    "actions": [
      {"click": "button.load-more"},
      {"wait_for": ".product-card:nth-child(40)"}
    ]
  }'

The response is the page after the last step, in the response_format you asked for, with two exceptions that switch it to the JSON envelope:

• A screenshot step. An image cannot travel in a markdown, text or HTML body, so a request with a screenshot step is answered with the JSON envelope whatever response_format says: the rendering under content (still in the format you asked for) and the images under screenshots, each under its step's label. The response carries a FORMAT_COERCED warning saying so. Send response_format: "json" to get the envelope without the warning.
• An evaluate step with return_value: true. Same rule: the value comes back under actions in the envelope, with the same warning.

The envelope's actions array echoes every step: index, label, ok, duration_ms, url_after (the page's URL when the step finished) and, for evaluate, return_value.

Step shape

Each step is an object with exactly one verb key, plus optional label, timeout_ms and on_error. Two verb keys in one step, or an unknown verb, is a 400 before anything runs.

String shorthand works for wait_for, wait_for_navigation, click and evaluate only. fill, select, scroll and screenshot need an object.

Shared keys, accepted on the step or inside the verb object (both spellings mean the same thing):

Use on_error: "skip" for steps that may legitimately find nothing, such as dismissing a cookie banner that only some visitors see.

Scrolling

scroll accepts one target. If you set more than one, to_bottom wins, then selector, then y.

• {"y": 1200} scrolls vertically by that many pixels (negative scrolls up). pixels is an alias. There is no horizontal scroll.
• {"selector": "#reviews"} brings that element to the centre of the viewport. No match fails the step.
• {"to_bottom": true} scrolls to the bottom repeatedly and stops when the page height stops growing, or when the step's timeout runs out. Use it for infinite-scroll feeds.

A scroll with none of these (or y: 0) is a 400.

evaluate and secret

evaluate is for side effects: removing an overlay, expanding collapsed sections, setting a value a widget ignores from fill. With {"expression": "…", "return_value": true} the expression's value, as JSON, is returned in the envelope's actions array under that step (return_value); add "await_promise": true for an async expression. Asking for a value switches the response to the JSON envelope, as a screenshot does.

"secret": true on fill redacts the value from logs, results, diagnostics and your request history. Set it on every password, token or one-time code.

Worked example: log in, scroll, extract

{
  "url": "https://example.com/login",
  "engine": "chromium",
  "actions": [
    {"click": "#accept-cookies", "on_error": "skip"},
    {"fill": {"selector": "input[name=email]", "value": "ada@example.com"}},
    {"fill": {"selector": "input[name=password]", "value": "correct-horse-battery", "secret": true}},
    {"click": {"selector": "button[type=submit]", "wait_for_navigation": true}, "label": "submit-login", "timeout_ms": 15000},
    {"wait_for": {"selector": ".orders-table", "timeout_ms": 15000}},
    {"scroll": {"to_bottom": true}, "timeout_ms": 20000},
    {"screenshot": {"full_page": true}, "label": "orders"}
  ],
  "extract": {
    "orders": {"selector": ".orders-table tbody tr", "kind": "list", "output": "text"},
    "next_page": "a.pagination-next@href"
  }
}

The response is the envelope: data.orders and data.next_page from the page as it stood after the last step, plus a screenshot labelled orders. To stay logged in across requests instead of logging in every time, run the login steps once with a session_id; see Sessions and logins.

Failure modes

Each step holds the browser for up to its own timeout, which is why the list is capped at 50.

Cost

Actions add no credits. You pay the engine that ran: obscura (js_render: true) 3 credits, chromium 8. A failed step costs 0.

Requests with actions are never served from or written to the cache, and they turn off the default image and font blocking so that the layout you click against is complete. See Credits.

Related

• Sessions and logins to keep cookies between requests
• Screenshots and PDF
• Structured data for the extract syntax
• CDP browser Coming soon