WebCrawlerAPI
WebCrawlerAPI provides crawling, page scraping, AI web agents, and monitored website feeds through a unified API.
Integration
- Vendor
- WebCrawlerAPI
- Category
- other
- Auth
- API_KEY
- Tools
- 10
- Composio slug
webcrawlerapi
Tools
- Create Crawl
Start a metered asynchronous crawl over a site and return its job ID. The crawl may visit and charge for up to items_limit pages; use GET_CRAWL_JOB to monitor it.
- Create Feed
Create a recurring website-change feed. Creation immediately schedules an initial crawl that may consume metered credits, and future scheduled runs may also consume credits; delete the feed when it is no longer needed.
- Delete Feeddestructive
Permanently cancel a feed so it cannot be resumed. The feed is removed from feed listings, but retained history may remain directly readable for 30 days.
- Get Crawl Job
Get a crawl job's status, configuration, per-page results, content URLs, errors, and observed costs. Respect recommended_pull_delay_ms when polling.
- Get Feed
Get one feed's configuration, lifecycle status, and recent run history, including per-run crawl counts and cost. Canceled feeds remain directly readable while retained by the provider.
- Get Organization Costs
Return current spendable balance plus request count and USD usage for a date range. The connected API key is used for both organization routes.
- List Feed Changes
Return one page of detected feed changes as structured JSON, with an opaque continuation cursor for older pages.
- List Feeds
List active and paused feeds for the connected organization, newest first. Logically canceled feeds are omitted.
- Scrape Page
Scrape one web page synchronously and return requested content or structured extraction. This consumes metered credits; supplying a prompt adds $0.002 per request.
- Set Feed State
Pause an active feed's future scheduled runs or resume a paused feed. Pausing does not stop a run that has already started, and resuming schedules future crawls that may consume metered credits.
Setup
- Create or open a Switchy Project.
- Open Integrations and connect WebCrawlerAPI.
- Authorise through api_key. Switchy stores the connection against your org.
- Everyone in the Project can now @-mention the WebCrawlerAPI tools in chat.
What teammates see: by default, memories from WebCrawlerAPI are scoped to the Project (PROJECT visibility) - you can mark any memory PRIVATE or share it ORG-wide.
Works well with
Top models
Compatibility data appears once enough Projects have used this MCP together with a given model.