LangChain document loader and tool

Two open-source packages add Spicrawl to LangChain, with the same classes in each language:

Both call POST /v1/scrape with response_format set to markdown, under your API key, with the same scopes, limits and credits as a direct API call.

To give an agent all 24 spicrawl_* tools instead of one scrape tool, connect the hosted MCP server: see LangChain and LangGraph.

Install

Create an API key at app.spicrawl.com and export it. Both packages read SPICRAWL_API_KEY.

export SPICRAWL_API_KEY="spicrawl_live_..."

pip install "git+https://github.com/OfficialSpicrawl/langchain-spicrawl.git"

pip install "langchain[openai]"

Load pages as documents

SpicrawlLoader takes one URL or a list. It scrapes them one after the other, in order, and returns one Document per page.

from langchain_spicrawl import SpicrawlLoader

loader = SpicrawlLoader(["https://example.com", "https://example.org"])
docs = loader.load()

print(docs[0].page_content)
print(docs[0].metadata)
# {'source': 'https://example.com', 'final_url': 'https://example.com/',
#  'target_status': 200, 'credits_charged': 1, 'engine': 'fetch',
#  'cache_state': 'hit', 'request_id': '01M4QYNXGPFF9VRGS64F0JE092'}

Python also has lazy_load(), aload() and alazy_load(). TypeScript has lazyLoad(), an async generator. Pass the documents to any text splitter and vector store, as with other LangChain loaders.

A page that needs JavaScript, or any other POST /v1/scrape field, goes in scrape_options (scrapeOptions in TypeScript). wait_for needs js_render.

loader = SpicrawlLoader(
    "https://example.com/pricing",
    scrape_options={"js_render": True, "wait_for": ".plans", "exclude_tags": ["nav", "footer"]},
    continue_on_failure=True,  # log a warning and skip a URL that fails
)

Loader parameters

The first argument is urls, required. The rest are keyword arguments in Python and the second argument's object fields in TypeScript.

Metadata

Both packages set the same keys, so documents from either can share one index. A key is left out when the API returned no value for it; title is left out when the page has no # heading.

Give an agent the scrape tool

SpicrawlScrapeTool takes no required arguments. The model sees two: url, and js_render, which it sets only when the first attempt came back empty because the page is built by JavaScript. Everything else is fixed by you when you create the tool. Set OPENAI_API_KEY for the model in these examples.

from langchain.agents import create_agent
from langchain_spicrawl import SpicrawlScrapeTool

tool = SpicrawlScrapeTool(max_chars=20_000)
agent = create_agent("openai:gpt-4.1", tools=[tool])

result = agent.invoke(
    {"messages": [{"role": "user", "content": "Read https://example.com/pricing and list the plans."}]}
)
print(result["messages"][-1].content)

Call the tool without an agent:

tool.invoke({"url": "https://example.com"})

Tool parameters

A page that comes back empty returns a message telling the model to call again with js_render set to true. The second call costs 3 credits.

Errors

A blocked target is an error, not content. When the site answers 4xx or 5xx (an access-denied page, a 404), the packages raise SpicrawlError with code set to TARGET_STATUS. The API itself returns such a page with HTTP 200 and an X-Target-Status header (see Response headers); the packages check it so a block page never reaches your index or your model.

Every SpicrawlError carries these fields:

from langchain_spicrawl import SpicrawlError, SpicrawlLoader

try:
    docs = SpicrawlLoader("https://example.com/pricing").load()
except SpicrawlError as err:
    print(err.code, err.status, err.request_id)

• The loader raises on the first failure. With continue_on_failure (continueOnFailure) it logs a warning and skips that URL, so the result can hold fewer documents than the URLs you passed.
• The tool in Python returns the error text to the model as the tool output, for example ERR::LIMIT::QUOTA_EXCEEDED: ..., so the agent can react instead of ending the run. In TypeScript the tool throws SpicrawlError, and createAgent passes the message back to the model.
• Retries. Only errors the API marks retryable are retried, up to max_retries times, after the Retry-After wait, or a short backoff when there is none. A wait longer than 30 seconds is raised instead. A dropped connection is never retried, because the scrape may already have run and been billed.

Credits

A cache hit (cache_state of hit) is billed at the price of the fetch that stored it, not for free. credits_charged in each document's metadata is what that page cost, and can differ from the table when a request is served by a stronger rendering tier. See Credits for the full price table.

Add up a load:

total = sum(d.metadata.get("credits_charged", 0) for d in docs)

The loader fetches one URL at a time. For thousands of URLs, use the batch API instead.

Troubleshooting

Source

• Python: github.com/OfficialSpicrawl/langchain-spicrawl (MIT)
• JavaScript and TypeScript: github.com/OfficialSpicrawl/langchain-spicrawl-js (MIT)

Related: Agent frameworks for the MCP route, Best practices for agents, Credits, Errors.