Dify

Use Spicrawl in Dify workflows and agents: install the Spicrawl plugin from its .difypkg file, authorize it with your API key, and call the Scrape, Extract and Batch scrape tools.

The Spicrawl plugin gives Dify three tools: Scrape (one page as markdown, text or HTML), Extract (named fields from one page as JSON) and Batch scrape (up to 100 URLs in one call). Use them as nodes in a workflow or as tools on an agent. Each call runs under your API key, with the same scopes, limits and credits as a direct API call.

Status

The plugin is submitted to the Dify Marketplace and is under review (langgenius/dify-plugins#3320). Until it is listed, install it from the package file below. It needs Dify 1.14.2 or later.

Install

Download the package

Download spicrawl-0.1.1.difypkg from the dify-v0.1.1 release.

Install it in Dify

Open Plugins → Install plugin → Local Package File and select the file. The plugin runtime must be able to reach https://api.spicrawl.com over HTTPS (port 443). Nothing else is needed.

Self-hosted Dify rejects unsigned packages by default

The package on GitHub is unsigned until the Dify Marketplace signs it, so a default self-hosted install fails with plugin verification has been enabled, and the plugin you want to install has a bad signature. Either wait for the Marketplace listing, or, on a Dify you control, add FORCE_VERIFYING_SIGNATURE=false to docker/.env and restart: cd docker && docker compose down && docker compose up -d. That setting lets Dify install any unsigned plugin, so use it only in an environment you control. On Dify Cloud, wait for the Marketplace listing.

Authorize with your API key

Create a key in app.spicrawl.com under API Keys (spicrawl_live_...). Give it the scrape scope, plus batch if you use Batch scrape. New keys carry both by default. In Dify, open the Spicrawl plugin, select To Authorize and paste the key. Dify checks it with a free call that spends no credits. See Authentication.

Tools

In a workflow, every parameter appears on the tool node. On an agent, the parameters marked Agent are chosen by the model, and the parameters marked You are set once in the tool's settings.

Scrape

Fetches one URL and returns it as markdown (default), plain text or HTML.

Parameter	Set by	Description
URL	Agent	The `http` or `https` page to fetch.
JavaScript rendering	Agent	Runs the page's JavaScript first. Turn it on when a normal scrape returns an empty or "loading" page. Costs 3 credits instead of 1.
Output format	You	Markdown, Plain text or HTML.
Main content only	You	Drops navigation, footers and sidebars (Markdown and Text). On by default.
Wait for selector	You	CSS selector to wait for before capture, such as `.price`. Turns on JavaScript rendering.
Include selectors, Exclude selectors	You	Comma-separated CSS selectors. Include keeps only matching parts; exclude removes them. Both cut tokens.
Proxy URL	You	Optional. Your own proxy (`http`, `https`, `socks5` or `socks5h`).

Outputs: text is the page content. json holds target_status (the site's HTTP status), credits, cache, request_id and any warning. If the site answers with an error status, text says so, so check target_status before you trust the content.

Extract

Returns named fields from one page as JSON. Provide Fields, Autoparse, or both.

Parameter	Set by	Description
URL	Agent	The page to extract from.
Fields (JSON)	Agent	A JSON object mapping a field name to a CSS selector. Append `@attribute` to read an attribute. A rule object such as `{"selector": ".tag", "kind": "list"}` returns every match.
Autoparse	Agent	Also returns the page's own metadata: JSON-LD, OpenGraph, Twitter Card, microdata and embedded app state. No selectors needed.
JavaScript rendering	Agent	Use it when selectors match nothing on a script-built page.
Wait for selector, Proxy URL	You	As in Scrape.

Fields (JSON)

{ "title": "h1", "price": ".price", "next_page": "a.next@href" }

Outputs: text is the extracted data as JSON. json holds data, empty_fields (selectors that matched nothing, the first sign a site changed its markup), target_status, credits and request_id. Extraction costs no more than a plain scrape. See Extract structured data.

Batch scrape

Scrapes up to 100 URLs as one job and waits for it to finish. Requires the batch scope.

Parameter	Set by	Description
URLs	Agent	One `http` or `https` URL per line, at most 100.
Job ID	Agent	Instead of URLs: collect the results of an earlier job that was still running.
Output format	You	Markdown, Plain text or HTML.
JavaScript rendering	You	Renders every page with JavaScript.
Max wait (seconds)	You	5 to 240, default 60. If the job is still running, the tool returns its Job ID; run the tool again with that Job ID. Results are kept for 72 hours.
Proxy URL	You	As in Scrape.

Outputs: text holds each page as a ## <url> section. json holds the job ID, status, progress and one entry per URL (status, target_status, error). Batch items cannot run extraction: use Extract per URL. For more than 100 URLs per call, use the API directly (Batch jobs).

Example workflow

A workflow that summarizes a page, and tells you when a field disappears:

Take the URL as input

Add a Start node with a text input named url.

Scrape the page

Add the Spicrawl Scrape tool. Set URL to the url input and Output format to Markdown. Turn on JavaScript rendering only if the first run comes back empty.

Summarize it

Add an LLM node. In its prompt, insert the Scrape node's text output:

LLM prompt

Summarize this page in five bullet points.

<page>
{Scrape / text}
</page>

Watch for layout changes (optional)

On a second branch, add the Spicrawl Extract tool with Fields {"title": "h1", "price": ".price"}. Add an IF/ELSE node that tests whether empty_fields in its json output is not empty, and route that case to an alert.

For an agent, add Scrape and Extract under the agent's tools and tell it when to escalate:

Agent instructions

Use the Spicrawl Scrape tool to read web pages. If the result is empty or looks like a
loading placeholder, call it again with JavaScript rendering on. Use Spicrawl Extract when
the user asks for specific fields such as a price or a title. Never retry on an
ERR::AUTH:: or ERR::LIMIT::QUOTA_EXCEEDED error.

Cost and errors

A plain fetch costs 1 credit and JavaScript rendering costs 3. Failures cost 0. Markdown conversion, selectors and extraction add nothing. See Credits.

Failures show the Spicrawl error code, its detail and a hint:

Code	What to do
`ERR::AUTH::INSUFFICIENT_SCOPE`	The key lacks a scope, usually `batch`. Create a key with it. Do not retry.
`ERR::AUTH::INVALID_KEY`	Re-enter the key under To Authorize.
`ERR::LIMIT::QUOTA_EXCEEDED`	Monthly credits are used up. Add credits. Do not retry.
`ERR::UPSTREAM::TIMEOUT`	The site was too slow. Retryable.

The full list is in Errors.

The plugin fetches whatever URL it is given. Spicrawl refuses private and reserved addresses, but in a workflow that takes the URL from untrusted users, still constrain that input. The optional Proxy URL can contain credentials, so treat the node configuration that holds it like a secret.