# n8n > Use Spicrawl in n8n with the n8n-nodes-spicrawl community node: scrape a URL to markdown, HTML, text, PDF or extracted fields, run batch jobs, and let an AI Agent node call it as a tool. Source: https://docs.spicrawl.com/agents/n8n The `n8n-nodes-spicrawl` community node adds a **Spicrawl** node to [n8n](https://n8n.io) with two operations, **Scrape URL** and **Batch Scrape**. It returns a page as markdown, HTML, text, a PDF or extracted fields, and an AI Agent node can call it as a tool. Each call runs under your API key, with the same scopes, limits and credits as a direct API call. > **Status:** The package is published on npm as [`n8n-nodes-spicrawl`](https://www.npmjs.com/package/n8n-nodes-spicrawl) and is under n8n's verification review. Until it is verified, install it by name on self-hosted n8n (below). n8n Cloud lists only verified community nodes, so Cloud users have to wait for the verification to finish. ## Install **Step 1: Install the package on self-hosted n8n** As the instance owner, open **Settings → Community Nodes → Install**, enter the package name, accept the risk prompt and select **Install**: ```text title="Package name" n8n-nodes-spicrawl ``` The **Spicrawl** node then appears in the node panel. **Step 2: Create an API key** In [app.spicrawl.com](https://app.spicrawl.com), open [**API Keys**](https://app.spicrawl.com/dashboard/keys) and create a key (`spicrawl_live_...`). Give it the `scrape` scope for Scrape URL and the `batch` scope for Batch Scrape. See [Authentication](https://docs.spicrawl.com/authentication.md). **Step 3: Add the credential** Add a Spicrawl node to a workflow, then select **Create new credential** and paste the key into **API Key**. n8n tests the credential with a free `GET /v1/requests?limit=1` call, which works with any valid key and spends no credits. A `401 ERR::AUTH::INVALID_KEY` means the key was copied wrongly. ## Example workflow Import this with **Workflow menu → Import from URL or clipboard**, select your credential on the Spicrawl node, and run it. It reads a page as markdown and extracts two fields: ```json title="spicrawl-scrape.json" { "nodes": [ { "name": "When clicking Test workflow", "type": "n8n-nodes-base.manualTrigger", "typeVersion": 1, "position": [0, 0], "parameters": {} }, { "name": "Spicrawl", "type": "n8n-nodes-spicrawl.spicrawl", "typeVersion": 1, "position": [220, 0], "parameters": { "operation": "scrape", "url": "https://example.com/products/42", "responseFormat": "markdown", "rendering": "fetch", "simplify": true, "options": { "extract": "{\n \"title\": \"h1\",\n \"price\": \".price\"\n}", "maxCost": 3 } } } ], "connections": { "When clicking Test workflow": { "main": [[{ "node": "Spicrawl", "type": "main", "index": 0 }]] } } } ``` The output item carries the page in `content` and the fields in `data`: ```json { "url": "https://example.com/products/42", "status": 200, "credits": 1, "content": "# Walnut Desk\n...", "data": { "title": "Walnut Desk", "price": "$349.00" }, "empty_fields": [] } ``` ## Scrape URL Fetches one page. It runs once per input item, so five input items make five calls. | Parameter | API field | Notes | | --------------- | ------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | URL | `url` | Required. Must start with `http://` or `https://`. | | Response Format | `response_format` | **Markdown** (default), **HTML**, **Text**, **JSON Envelope** or **PDF**. | | Rendering | `js_render`, `mode` | **Fetch** (default, plain HTTP, 1 credit), **JavaScript** (3 credits) or **Auto** (tries a plain fetch and escalates only when the page needs it; you pay for the tier that served it). | | Simplify | | On by default. Returns the most useful fields only. Turn it off for the complete response, including `credits_remaining` and `cache_state`. | Add these under **Options**: | Option | API field | Notes | | --------------------------------------------------- | -------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | | Auto-Parse Page Metadata | `autoparse` | JSON-LD, OpenGraph, Twitter Card and microdata, returned under `data`. | | Block Resources | `block_resources` | Image, Font, Media, Script, Stylesheet. Needs JavaScript or Auto rendering. | | Bypass Cache | `cache` | Sends `cache: false`. A cache hit is billed like the fetch that stored it. | | Cache TTL (Seconds) | `cache_ttl` | Cannot be combined with Bypass Cache. | | Custom Headers | `custom_headers` | Sent to the target site. | | Extract Fields (JSON) | `extract` | See [Extract fields](#extract-fields). | | Main Content Only | `main_content_only` | Strips navigation, footers and sidebars from markdown and text. | | Max Cost (Credits) | `max_cost` | Refuses the request at 0 credits if its dearest outcome costs more. Set it on every node an agent can call. | | Proxy URL | `proxy` | Your own proxy. Residential proxies work best on protected sites. | | Return Links | `links` | Every absolute link on the page, under `links`. | | Screenshot, Screenshot Format, Screenshot Full Page | `screenshot`, `screenshot_format`, `screenshot_fullpage` | See [Screenshots and PDFs](#screenshots-and-pdfs). Format is PNG, JPEG or WebP. | | Wait After Load (Ms) | `wait` | 1 to 30000. Needs JavaScript or Auto rendering. | | Wait For Selector | `wait_for` | CSS selector. Needs JavaScript or Auto rendering. | Check `status` in the output: a blocked or missing page is still a successful node run, and `status` is what the target site answered with. ### Extract fields Set **Extract Fields (JSON)** to a map of field name to CSS selector, or to a JSON Schema whose properties each carry a `selector`. The result is returned under `data`, and `empty_fields` lists every selector that matched nothing, which is the first sign a site changed its markup. Extraction costs no extra credits. For script-built pages, set Rendering to JavaScript and add **Wait For Selector**. ```json title="Extract Fields (JSON)" { "title": "h1", "price": ".price", "images": { "selector": ".gallery img", "kind": "list", "output": "attr", "attribute": "src" }, "next_page": "a.next@href" } ``` `a.next@href` reads an attribute; `"kind": "list"` returns every match. See [Extract structured data](https://docs.spicrawl.com/guides/structured-data.md) for the schema form and type coercion. ### Screenshots and PDFs Binary output is attached to the item, not placed in `json`: | You set | Binary property | File name | | ------------------------ | ------------------------------------------- | ------------------------------------ | | Response Format: **PDF** | `data` | `page.pdf` | | Option **Screenshot** | `screenshot` (`screenshot_2`, ... for more) | `screenshot.png`, `.jpeg` or `.webp` | Pass the property to a **Write Binary File**, **Google Drive** or **Send Email** node. Both use full-browser rendering, which costs more (8 credits); see [Credits](https://docs.spicrawl.com/credits.md) and [Screenshots and PDF](https://docs.spicrawl.com/guides/screenshots-and-pdf.md). A screenshot or PDF is always binary, whatever Simplify is set to. ## Batch Scrape Submits a list of URLs as one job, waits for it to finish, then returns one item per URL. It runs once for all input items, not once per item. Requires the `batch` scope. | Parameter | Notes | | --------------- | -------------------------------------------------------------------------------------------------------------------------------------------- | | URL Source | **Text List** (one URL per line) or **Input Items** (one URL from each input item, default expression `{{ $json.url }}`). Up to 10,000 URLs. | | Response Format | **Markdown** (default), **HTML** or **Text**. | | Rendering | **Fetch** (default) or **JavaScript**. | | Simplify | On by default. | Options: Block Resources, Concurrency (1 to 50, default 10), Credit Budget, Custom Headers, Failure Threshold, Job Name, Main Content Only, Max Attempts, Max Cost Per Page (Credits), Max Wait (Minutes, default 30), Poll Interval (Seconds), Proxy URL, Wait After Load (Ms) and Wait For Selector. Each output item has `seq`, `url`, `status` (`succeeded`, `failed`, `cancelled` or `skipped`), `http_status`, `content`, `credits` and, for a failed URL, `error.code` and `error.retryable`. Failed URLs are returned too, so filter on `status` with an **IF** or **Filter** node. If the job outlives **Max Wait**, the node stops with an error naming the job ID; the job keeps running, finished pages are still charged and results stay readable for 72 hours. > **Warning:** Batch items cannot run extraction, screenshots, PDFs or Auto rendering. The API refuses them with a `400` that names each field, and nothing is charged. Use Scrape URL for those. See [Batch jobs](https://docs.spicrawl.com/guides/batch.md). ## Use Spicrawl as an AI Agent tool The node is marked usable as a tool, so an **AI Agent** node can call both operations and fill in the URL and options itself. **Step 1: Allow community nodes as tools** On self-hosted n8n, set this environment variable and restart n8n. Without it, n8n does not offer community nodes as agent tools. ```bash N8N_COMMUNITY_PACKAGES_ALLOW_TOOL_USAGE=true ``` **Step 2: Add Spicrawl to the agent** In the **AI Agent** node, select the **+** under **Tool** and choose **Spicrawl**. Keep Response Format on Markdown, which is much smaller than HTML for the model, and set **Max Cost (Credits)** so one tool call cannot spend more than you intend. ## Troubleshooting | Symptom | What it means | What to do | | ---------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------- | -------------------------------------------------------------------- | | Spicrawl does not appear in n8n Cloud | Cloud lists only verified community nodes. | Wait for verification, or use self-hosted n8n. | | Spicrawl is missing from the agent's tool list | Tool usage for community nodes is off. | Set `N8N_COMMUNITY_PACKAGES_ALLOW_TOOL_USAGE=true` and restart. | | `ERR::AUTH::INVALID_KEY` or `ERR::AUTH::MISSING_KEY` | The key is wrong or empty. | Copy it again into the credential. | | `ERR::AUTH::INSUFFICIENT_SCOPE` | The key lacks the scope for this operation. | Use a key with `scrape` (Scrape URL) or `batch` (Batch Scrape). | | `ERR::REQUEST::INCOMPATIBLE_FLAGS` | An option needs another one, for example Wait For Selector needs JavaScript or Auto rendering. | Change Rendering. | | `ERR::LIMIT::QUOTA_EXCEEDED` (402) | Your monthly credits cannot cover the request. | Submit fewer URLs, set Max Cost, or add credits. Do not retry. | | `ERR::LIMIT::RATE_LIMITED` or `ERR::LIMIT::CONCURRENCY_EXCEEDED` (429) | Too many requests per second, or too many in flight. Retryable and free. | Turn on **Retry On Fail** in the node settings, or use Batch Scrape. | Errors appear in n8n with their stable code, the API's hint and the request ID. Use **Retry On Fail** only for errors marked retryable: scraping is not idempotent, so a repeated successful call is billed again, even when the cache serves it. The full list is in [Errors](https://docs.spicrawl.com/errors.md). ## Links * [n8n-nodes-spicrawl on GitHub](https://github.com/OfficialSpicrawl/n8n-nodes-spicrawl) (source, issues, changelog) * [n8n-nodes-spicrawl on npm](https://www.npmjs.com/package/n8n-nodes-spicrawl) * [n8n: install community nodes](https://docs.n8n.io/integrations/community-nodes/installation/) * [Credits](https://docs.spicrawl.com/credits.md), [Batch jobs](https://docs.spicrawl.com/guides/batch.md), [Extract structured data](https://docs.spicrawl.com/guides/structured-data.md)