# Dify

> Use Spicrawl in Dify workflows and agents: install the Spicrawl plugin from its .difypkg file, authorize it with your API key, and call the Scrape, Extract and Batch scrape tools.

Source: https://docs.spicrawl.com/agents/dify

The Spicrawl plugin gives Dify three tools: **Scrape** (one page as markdown, text or HTML), **Extract** (named fields from one page as JSON) and **Batch scrape** (up to 100 URLs in one call). Use them as nodes in a workflow or as tools on an agent. Each call runs under your API key, with the same scopes, limits and credits as a direct API call.

> **Status:** The plugin is submitted to the Dify Marketplace and is under review ([langgenius/dify-plugins#3320](https://github.com/langgenius/dify-plugins/pull/3320)). Until it is listed, install it from the package file below. It needs Dify 1.14.2 or later.

## Install

**Step 1: Download the package**

Download `spicrawl-0.1.1.difypkg` from the [`dify-v0.1.1` release](https://github.com/OfficialSpicrawl/agent-plugins/releases/tag/dify-v0.1.1).

**Step 2: Install it in Dify**

Open **Plugins → Install plugin → Local Package File** and select the file. The plugin runtime must be able to reach `https://api.spicrawl.com` over HTTPS (port 443). Nothing else is needed.

> **Self-hosted Dify rejects unsigned packages by default:** The package on GitHub is unsigned until the Dify Marketplace signs it, so a default self-hosted install fails with `plugin verification has been enabled, and the plugin you want to install has a bad signature`. Either wait for the Marketplace listing, or, on a Dify you control, add `FORCE_VERIFYING_SIGNATURE=false` to `docker/.env` and restart: `cd docker && docker compose down && docker compose up -d`. That setting lets Dify install any unsigned plugin, so use it only in an environment you control. On Dify Cloud, wait for the Marketplace listing.

**Step 3: Authorize with your API key**

Create a key in [app.spicrawl.com](https://app.spicrawl.com) under [**API Keys**](https://app.spicrawl.com/dashboard/keys) (`spicrawl_live_...`). Give it the `scrape` scope, plus `batch` if you use Batch scrape. New keys carry both by default. In Dify, open the Spicrawl plugin, select **To Authorize** and paste the key. Dify checks it with a free call that spends no credits. See [Authentication](https://docs.spicrawl.com/authentication.md).

## Tools

In a workflow, every parameter appears on the tool node. On an agent, the parameters marked **Agent** are chosen by the model, and the parameters marked **You** are set once in the tool's settings.

### Scrape

Fetches one URL and returns it as markdown (default), plain text or HTML.

| Parameter                            | Set by | Description                                                                                                                         |
| ------------------------------------ | ------ | ----------------------------------------------------------------------------------------------------------------------------------- |
| URL                                  | Agent  | The `http` or `https` page to fetch.                                                                                                |
| JavaScript rendering                 | Agent  | Runs the page's JavaScript first. Turn it on when a normal scrape returns an empty or "loading" page. Costs 3 credits instead of 1. |
| Output format                        | You    | Markdown, Plain text or HTML.                                                                                                       |
| Main content only                    | You    | Drops navigation, footers and sidebars (Markdown and Text). On by default.                                                          |
| Wait for selector                    | You    | CSS selector to wait for before capture, such as `.price`. Turns on JavaScript rendering.                                           |
| Include selectors, Exclude selectors | You    | Comma-separated CSS selectors. Include keeps only matching parts; exclude removes them. Both cut tokens.                            |
| Proxy URL                            | You    | Optional. Your own proxy (`http`, `https`, `socks5` or `socks5h`).                                                                  |

Outputs: `text` is the page content. `json` holds `target_status` (the site's HTTP status), `credits`, `cache`, `request_id` and any `warning`. If the site answers with an error status, `text` says so, so check `target_status` before you trust the content.

### Extract

Returns named fields from one page as JSON. Provide Fields, Autoparse, or both.

| Parameter                    | Set by | Description                                                                                                                                                                       |
| ---------------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| URL                          | Agent  | The page to extract from.                                                                                                                                                         |
| Fields (JSON)                | Agent  | A JSON object mapping a field name to a CSS selector. Append `@attribute` to read an attribute. A rule object such as `{"selector": ".tag", "kind": "list"}` returns every match. |
| Autoparse                    | Agent  | Also returns the page's own metadata: JSON-LD, OpenGraph, Twitter Card, microdata and embedded app state. No selectors needed.                                                    |
| JavaScript rendering         | Agent  | Use it when selectors match nothing on a script-built page.                                                                                                                       |
| Wait for selector, Proxy URL | You    | As in Scrape.                                                                                                                                                                     |

```json title="Fields (JSON)"
{ "title": "h1", "price": ".price", "next_page": "a.next@href" }
```

Outputs: `text` is the extracted `data` as JSON. `json` holds `data`, `empty_fields` (selectors that matched nothing, the first sign a site changed its markup), `target_status`, `credits` and `request_id`. Extraction costs no more than a plain scrape. See [Extract structured data](https://docs.spicrawl.com/guides/structured-data.md).

### Batch scrape

Scrapes up to 100 URLs as one job and waits for it to finish. Requires the `batch` scope.

| Parameter            | Set by | Description                                                                                                                                         |
| -------------------- | ------ | --------------------------------------------------------------------------------------------------------------------------------------------------- |
| URLs                 | Agent  | One `http` or `https` URL per line, at most 100.                                                                                                    |
| Job ID               | Agent  | Instead of URLs: collect the results of an earlier job that was still running.                                                                      |
| Output format        | You    | Markdown, Plain text or HTML.                                                                                                                       |
| JavaScript rendering | You    | Renders every page with JavaScript.                                                                                                                 |
| Max wait (seconds)   | You    | 5 to 240, default 60. If the job is still running, the tool returns its Job ID; run the tool again with that Job ID. Results are kept for 72 hours. |
| Proxy URL            | You    | As in Scrape.                                                                                                                                       |

Outputs: `text` holds each page as a `## <url>` section. `json` holds the job ID, status, progress and one entry per URL (`status`, `target_status`, `error`). Batch items cannot run extraction: use Extract per URL. For more than 100 URLs per call, use the API directly ([Batch jobs](https://docs.spicrawl.com/guides/batch.md)).

## Example workflow

A workflow that summarizes a page, and tells you when a field disappears:

**Step 1: Take the URL as input**

Add a **Start** node with a text input named `url`.

**Step 2: Scrape the page**

Add the Spicrawl **Scrape** tool. Set **URL** to the `url` input and **Output format** to Markdown. Turn on **JavaScript rendering** only if the first run comes back empty.

**Step 3: Summarize it**

Add an **LLM** node. In its prompt, insert the Scrape node's `text` output:

```text title="LLM prompt"
Summarize this page in five bullet points.

<page>
{Scrape / text}
</page>
```

**Step 4: Watch for layout changes (optional)**

On a second branch, add the Spicrawl **Extract** tool with Fields `{"title": "h1", "price": ".price"}`. Add an **IF/ELSE** node that tests whether `empty_fields` in its `json` output is not empty, and route that case to an alert.

For an agent, add **Scrape** and **Extract** under the agent's tools and tell it when to escalate:

```text title="Agent instructions"
Use the Spicrawl Scrape tool to read web pages. If the result is empty or looks like a
loading placeholder, call it again with JavaScript rendering on. Use Spicrawl Extract when
the user asks for specific fields such as a price or a title. Never retry on an
ERR::AUTH:: or ERR::LIMIT::QUOTA_EXCEEDED error.
```

## Cost and errors

A plain fetch costs 1 credit and JavaScript rendering costs 3. Failures cost 0. Markdown conversion, selectors and extraction add nothing. See [Credits](https://docs.spicrawl.com/credits.md).

Failures show the Spicrawl error code, its detail and a hint:

| Code                            | What to do                                                                  |
| ------------------------------- | --------------------------------------------------------------------------- |
| `ERR::AUTH::INSUFFICIENT_SCOPE` | The key lacks a scope, usually `batch`. Create a key with it. Do not retry. |
| `ERR::AUTH::INVALID_KEY`        | Re-enter the key under **To Authorize**.                                    |
| `ERR::LIMIT::QUOTA_EXCEEDED`    | Monthly credits are used up. Add credits. Do not retry.                     |
| `ERR::UPSTREAM::TIMEOUT`        | The site was too slow. Retryable.                                           |

The full list is in [Errors](https://docs.spicrawl.com/errors.md).

> **Warning:** The plugin fetches whatever URL it is given. Spicrawl refuses private and reserved addresses, but in a workflow that takes the URL from untrusted users, still constrain that input. The optional Proxy URL can contain credentials, so treat the node configuration that holds it like a secret.

## Links

* [Plugin source](https://github.com/OfficialSpicrawl/agent-plugins/tree/main/dify) (README, tool definitions, privacy policy)
* [`dify-v0.1.1` release](https://github.com/OfficialSpicrawl/agent-plugins/releases/tag/dify-v0.1.1)
* [Credits](https://docs.spicrawl.com/credits.md), [Batch jobs](https://docs.spicrawl.com/guides/batch.md), [Extract structured data](https://docs.spicrawl.com/guides/structured-data.md)