Skip to main content

Web Crawl Tool

Multi-page web crawling tool for exploring entire websites and subdomains.

Quick Start​

import { WebCrawlTool } from '@bubblelab/bubble-core';
import { CredentialType } from '@bubblelab/shared-schemas';
const result = await new WebCrawlTool({

  url: 'example', // The root URL to crawl and extract content from
  credentials: {
    [CredentialType.FIRECRAWL_API_KEY]: process.env.FIRECRAWL_API_KEY as string,
  },
}).action();

Operation Details​

execute​

url string required
The root URL to crawl and extract content from
format 'markdown'
Output format for crawled content
onlyMainContent boolean
Extract only main content, filtering out navigation/footers
maxPages number
Maximum number of pages to crawl
crawlDepth number
Maximum depth to crawl
includePaths string[]
URL patterns to include in crawl (regex patterns), Example: ["^/blog/.$", "^/docs/.$"]
excludePaths string[]
URL patterns to exclude from crawl (regex patterns), ["^/admin/.$", "^/private/.$"]
waitFor number
Time to wait for dynamic content in milliseconds
credentials Record<CredentialType,string>
Required credentials including FIRECRAWL_API_KEY