# LangChain document loader and tool

> Load web pages as LangChain Documents with SpicrawlLoader and let an agent read a URL with SpicrawlScrapeTool, using the open-source Python and JavaScript packages.

Source: https://docs.spicrawl.com/agents/langchain

Two open-source packages add Spicrawl to LangChain, with the same classes in each language:

| Class                | Use it to                                                                         |
| -------------------- | --------------------------------------------------------------------------------- |
| `SpicrawlLoader`     | Turn URLs into `Document`s (markdown content plus metadata) for RAG and indexing. |
| `SpicrawlScrapeTool` | Let an agent open a URL and read it. The tool is named `spicrawl_scrape`.         |

Both call `POST /v1/scrape` with `response_format` set to `markdown`, under your API key, with the same scopes, limits and credits as a direct API call.

> **Install from GitHub:** The packages are not on PyPI or npm yet. Install them from their GitHub repos, as below.

To give an agent all 24 `spicrawl_*` tools instead of one scrape tool, connect the hosted MCP server: see [LangChain and LangGraph](https://docs.spicrawl.com/agents/frameworks.md#langchain-and-langgraph).

## Install

Create an API key at [app.spicrawl.com](https://app.spicrawl.com/dashboard/keys) and export it. Both packages read `SPICRAWL_API_KEY`.

```bash
export SPICRAWL_API_KEY="spicrawl_live_..."
```

**Python**

Python 3.10 or later, with `langchain-core` 1.x.

```bash
pip install "git+https://github.com/OfficialSpicrawl/langchain-spicrawl.git"
```

The agent example below also needs a model package:

```bash
pip install "langchain[openai]"
```

Import from `langchain_spicrawl`.

**JavaScript / TypeScript**

The package is `@spicrawl/langchain`, with `@langchain/core` 1.x as a peer dependency. It runs on Node.js 20 or later; building it needs Node.js 22.

The repo has no `prepare` script and `dist/` is not committed, so `npm install github:OfficialSpicrawl/langchain-spicrawl-js` would install it unbuilt. Clone, build, pack, then install the tarball:

```bash
git clone https://github.com/OfficialSpicrawl/langchain-spicrawl-js.git
cd langchain-spicrawl-js
npm ci && npm run build && npm pack

cd /path/to/your/project
npm install /path/to/langchain-spicrawl-js/spicrawl-langchain-0.1.0.tgz @langchain/core
```

Install the tarball, not the folder. A folder install symlinks the clone, whose own `node_modules` carries a second copy of `@langchain/core`, and two copies break LangChain's `instanceof` checks. The tarball name is `spicrawl-langchain-<version>.tgz`, with the version from the clone's `package.json`. After you pull changes, run `npm run build && npm pack` again and reinstall the tarball. The agent example below also needs `langchain` and a model package such as `@langchain/openai`.

## Load pages as documents

`SpicrawlLoader` takes one URL or a list. It scrapes them one after the other, in order, and returns one `Document` per page.

```python title="Python"
from langchain_spicrawl import SpicrawlLoader

loader = SpicrawlLoader(["https://example.com", "https://example.org"])
docs = loader.load()

print(docs[0].page_content)
print(docs[0].metadata)
# {'source': 'https://example.com', 'final_url': 'https://example.com/',
#  'target_status': 200, 'credits_charged': 1, 'engine': 'fetch',
#  'cache_state': 'hit', 'request_id': '01M4QYNXGPFF9VRGS64F0JE092'}
```

```ts title="TypeScript"
import { SpicrawlLoader } from "@spicrawl/langchain";

const loader = new SpicrawlLoader(["https://example.com", "https://example.org"]);
const docs = await loader.load();

console.log(docs[0].pageContent);
console.log(docs[0].metadata);
// {
//   source: 'https://example.com', final_url: 'https://example.com/', target_status: 200,
//   credits_charged: 1, engine: 'fetch', cache_state: 'hit', request_id: '01M4QYNXGPFF9VRGS64F0JE092'
// }
```

Python also has `lazy_load()`, `aload()` and `alazy_load()`. TypeScript has `lazyLoad()`, an async generator. Pass the documents to any text splitter and vector store, as with other LangChain loaders.

A page that needs JavaScript, or any other `POST /v1/scrape` field, goes in `scrape_options` (`scrapeOptions` in TypeScript). `wait_for` needs `js_render`.

```python title="Python"
loader = SpicrawlLoader(
    "https://example.com/pricing",
    scrape_options={"js_render": True, "wait_for": ".plans", "exclude_tags": ["nav", "footer"]},
    continue_on_failure=True,  # log a warning and skip a URL that fails
)
```

```ts title="TypeScript"
const loader = new SpicrawlLoader("https://example.com/pricing", {
  scrapeOptions: { js_render: true, wait_for: ".plans", exclude_tags: ["nav", "footer"] },
  continueOnFailure: true, // console.warn and skip a URL that fails
});
```

### Loader parameters

The first argument is `urls`, required. The rest are keyword arguments in Python and the second argument's object fields in TypeScript.

| Python                | TypeScript          | Default                                               | Meaning                                                                                                                          |
| --------------------- | ------------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |
| `api_key`             | `apiKey`            | `$SPICRAWL_API_KEY`                                   | Spicrawl API key. Without one the constructor raises.                                                                            |
| `base_url`            | `baseUrl`           | `$SPICRAWL_BASE_URL`, else `https://api.spicrawl.com` | API base URL.                                                                                                                    |
| `scrape_options`      | `scrapeOptions`     | `{}`                                                  | Extra [`POST /v1/scrape`](https://docs.spicrawl.com/api-reference/scrape/scrape-post.md) fields sent with every URL. `response_format` is always `markdown`. |
| `continue_on_failure` | `continueOnFailure` | `False` / `false`                                     | Log a warning and skip a failed URL instead of raising.                                                                          |
| `timeout`             | `timeout`           | `120` seconds / `120000` milliseconds                 | Wait for each response.                                                                                                          |
| `max_retries`         | `maxRetries`        | `2`                                                   | Retries of errors the API marks `retryable`.                                                                                     |

### Metadata

Both packages set the same keys, so documents from either can share one index. A key is left out when the API returned no value for it; `title` is left out when the page has no `#` heading.

| Key               | Value                                                                     |
| ----------------- | ------------------------------------------------------------------------- |
| `source`          | The URL you passed.                                                       |
| `title`           | The first `#` heading in the markdown.                                    |
| `final_url`       | The URL after redirects.                                                  |
| `target_status`   | The site's HTTP status.                                                   |
| `credits_charged` | Credits this page cost.                                                   |
| `engine`          | The rendering tier that served the page: `fetch`, `js` or `browser`.      |
| `cache_state`     | `hit`, `miss` or `bypass`.                                                |
| `request_id`      | Quote it to Spicrawl support, or look it up with `GET /v1/requests/{id}`. |

## Give an agent the scrape tool

`SpicrawlScrapeTool` takes no required arguments. The model sees two: `url`, and `js_render`, which it sets only when the first attempt came back empty because the page is built by JavaScript. Everything else is fixed by you when you create the tool. Set `OPENAI_API_KEY` for the model in these examples.

```python title="Python"
from langchain.agents import create_agent
from langchain_spicrawl import SpicrawlScrapeTool

tool = SpicrawlScrapeTool(max_chars=20_000)
agent = create_agent("openai:gpt-4.1", tools=[tool])

result = agent.invoke(
    {"messages": [{"role": "user", "content": "Read https://example.com/pricing and list the plans."}]}
)
print(result["messages"][-1].content)
```

```ts title="TypeScript"
import { createAgent } from "langchain";
import { SpicrawlScrapeTool } from "@spicrawl/langchain";

const agent = createAgent({
  model: "openai:gpt-4.1",
  tools: [new SpicrawlScrapeTool({ maxChars: 20_000 })],
});

const result = await agent.invoke({
  messages: [{ role: "user", content: "Read https://example.com/pricing and list the plans." }],
});
console.log(result.messages.at(-1)?.content);
```

Call the tool without an agent:

```python title="Python"
tool.invoke({"url": "https://example.com"})
```

```ts title="TypeScript"
await new SpicrawlScrapeTool().invoke({ url: "https://example.com" });
```

### Tool parameters

| Python           | TypeScript      | Default                                               | Meaning                                                                                                                    |
| ---------------- | --------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `api_key`        | `apiKey`        | `$SPICRAWL_API_KEY`                                   | Spicrawl API key.                                                                                                          |
| `base_url`       | `baseUrl`       | `$SPICRAWL_BASE_URL`, else `https://api.spicrawl.com` | API base URL.                                                                                                              |
| `scrape_options` | `scrapeOptions` | `{}`                                                  | Extra `POST /v1/scrape` fields for every call. `response_format` is always `markdown`.                                     |
| `max_chars`      | `maxChars`      | `30000`                                               | The page is cut to this many characters before it reaches the model, followed by `[truncated: showing N of M characters]`. |
| `timeout`        | `timeout`       | `120` seconds / `120000` milliseconds                 | Wait for each response.                                                                                                    |
| `max_retries`    | `maxRetries`    | `2`                                                   | Retries of errors the API marks `retryable`.                                                                               |

A page that comes back empty returns a message telling the model to call again with `js_render` set to true. The second call costs 3 credits.

## Errors

A blocked target is an error, not content. When the site answers 4xx or 5xx (an access-denied page, a 404), the packages raise `SpicrawlError` with `code` set to `TARGET_STATUS`. The API itself returns such a page with HTTP 200 and an `X-Target-Status` header (see [Response headers](https://docs.spicrawl.com/response-headers.md)); the packages check it so a block page never reaches your index or your model.

Every `SpicrawlError` carries these fields:

| Python        | TypeScript   | Meaning                                                                                                                                                                                                                 |
| ------------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `code`        | `code`       | The API's error code, such as `ERR::LIMIT::RATE_LIMITED` or `ERR::LIMIT::QUOTA_EXCEEDED`, or one of `TARGET_STATUS` (the site answered 4xx/5xx) and `CONNECTION` (no response). All API codes are in [Errors](https://docs.spicrawl.com/errors.md). |
| `status`      | `status`     | The HTTP status of the Spicrawl response.                                                                                                                                                                               |
| `retryable`   | `retryable`  | Whether the API marked the error as safe to retry.                                                                                                                                                                      |
| `retry_after` | `retryAfter` | Seconds the API asked you to wait, if any.                                                                                                                                                                              |
| `request_id`  | `requestId`  | Quote it to Spicrawl support.                                                                                                                                                                                           |

```python title="Python"
from langchain_spicrawl import SpicrawlError, SpicrawlLoader

try:
    docs = SpicrawlLoader("https://example.com/pricing").load()
except SpicrawlError as err:
    print(err.code, err.status, err.request_id)
```

```ts title="TypeScript"
import { SpicrawlError, SpicrawlLoader } from "@spicrawl/langchain";

try {
  const docs = await new SpicrawlLoader("https://example.com/pricing").load();
} catch (err) {
  if (err instanceof SpicrawlError) console.error(err.code, err.status, err.requestId);
  else throw err;
}
```

* **The loader** raises on the first failure. With `continue_on_failure` (`continueOnFailure`) it logs a warning and skips that URL, so the result can hold fewer documents than the URLs you passed.
* **The tool in Python** returns the error text to the model as the tool output, for example `ERR::LIMIT::QUOTA_EXCEEDED: ...`, so the agent can react instead of ending the run. **In TypeScript** the tool throws `SpicrawlError`, and `createAgent` passes the message back to the model.
* **Retries.** Only errors the API marks `retryable` are retried, up to `max_retries` times, after the `Retry-After` wait, or a short backoff when there is none. A wait longer than 30 seconds is raised instead. A dropped connection is never retried, because the scrape may already have run and been billed.

## Credits

| Request                                 | Credits |
| --------------------------------------- | ------- |
| Plain fetch (default)                   | 1       |
| `js_render`                             | 3       |
| Any failure, including a blocked target | 0       |

A cache hit (`cache_state` of `hit`) is billed at the price of the fetch that stored it, not for free. `credits_charged` in each document's metadata is what that page cost, and can differ from the table when a request is served by a stronger rendering tier. See [Credits](https://docs.spicrawl.com/credits.md) for the full price table.

Add up a load:

```python title="Python"
total = sum(d.metadata.get("credits_charged", 0) for d in docs)
```

```ts title="TypeScript"
const total = docs.reduce((sum, d) => sum + (d.metadata.credits_charged ?? 0), 0);
```

The loader fetches one URL at a time. For thousands of URLs, use the [batch API](https://docs.spicrawl.com/guides/batch.md) instead.

## Troubleshooting

| Symptom                                                                                                                                                     | Cause                                                                         | Fix                                                                                                                                       |
| ----------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------- |
| A missing-API-key error when you create a loader or tool (`No API key: pass api_key=... or set the SPICRAWL_API_KEY environment variable.` from the loader) | The key is not set in the process that runs your code.                        | Export `SPICRAWL_API_KEY`, or pass `api_key` (`apiKey`).                                                                                  |
| `Cannot find module '@spicrawl/langchain'` or a missing `dist/` file after the TypeScript install                                                           | The clone was not built, or you installed the folder instead of the tarball.  | Run `npm ci && npm run build && npm pack` in the clone, then install the `.tgz`.                                                          |
| `SpicrawlError` with `code` `TARGET_STATUS`                                                                                                                 | The site answered 4xx or 5xx. This is a block or a missing page, not content. | The message names the site's status. A 404 or 410 means the URL is wrong; for a 403 or a challenge page see [Anti-bot](https://docs.spicrawl.com/guides/anti-bot.md). |
| `ERR::LIMIT::QUOTA_EXCEEDED`                                                                                                                                | The monthly credit allowance is used up. It is not retryable.                 | Wait for the reset date in the error, or raise the limit. See [Credits](https://docs.spicrawl.com/credits.md).                                                        |
| The loader returns fewer documents than URLs                                                                                                                | `continue_on_failure` skipped failed URLs.                                    | Read the logged warnings, or turn the option off to see the error.                                                                        |
| The tool reports `returned no readable content`                                                                                                             | The page is built by JavaScript.                                              | Let the model call again with `js_render` true, or set `js_render` in `scrape_options` for every call (3 credits each).                   |

## Source

* Python: [github.com/OfficialSpicrawl/langchain-spicrawl](https://github.com/OfficialSpicrawl/langchain-spicrawl) (MIT)
* JavaScript and TypeScript: [github.com/OfficialSpicrawl/langchain-spicrawl-js](https://github.com/OfficialSpicrawl/langchain-spicrawl-js) (MIT)

Related: [Agent frameworks](https://docs.spicrawl.com/agents/frameworks.md) for the MCP route, [Best practices for agents](https://docs.spicrawl.com/agents/best-practices.md), [Credits](https://docs.spicrawl.com/credits.md), [Errors](https://docs.spicrawl.com/errors.md).