WebCrawlerAPI

@WebCrawlerAPI start a crawl of example.com/docs with a 50-page limit and tell me when it finishes

I've started crawl job `crawl_8x2mK9pL` on example.com/docs with a 50-page limit. The job is currently running and has visited 12 pages so far. I'll check back in 30 seconds as recommended by the API's polling delay. Once complete, I can retrieve the full page list, extracted content URLs, and any errors encountered. Note that each page visited counts against your metered credit balance.

This example shows the async crawl workflow: CREATE_CRAWL returns a job ID immediately, then GET_CRAWL_JOB polls for completion. The MCP requires an API key with crawl permissions. Be aware that crawls consume metered credits per page visited, and the job may hit the items_limit before finishing the entire site.

@WebCrawlerAPI create a daily feed watching blog.example.com/releases for new posts, then show me what changed

I've created feed `feed_3nQ7wRt` monitoring blog.example.com/releases with daily checks at 09:00 UTC. The initial crawl has completed and found 8 pages. I've also retrieved the first page of detected changes: 2 new posts appeared since the baseline ('v2.1.0 Release Notes' on Jan 15, 'Security Update 2.0.3' on Jan 12). The feed will continue running daily until you delete it, and each scheduled run will consume additional metered credits.

This example demonstrates both CREATE_FEED and LIST_FEED_CHANGES. Feeds are recurring jobs that charge credits on every scheduled run, not just at creation. You must explicitly DELETE_FEED to stop billing. Useful for monitoring competitor sites, documentation updates, or news sources, but remember the ongoing cost.

@WebCrawlerAPI check my organization's crawl spending for the last 7 days and estimate if I can afford a 200-page crawl

Your organization has used $4.32 across 1,847 page requests in the last 7 days, leaving a spendable balance of $45.68. At your current average cost of ~$0.0023 per page, a 200-page crawl would cost approximately $0.46, well within your remaining budget. However, actual costs vary by page complexity and network conditions, so the real charge may differ. I recommend starting the crawl and monitoring GET_CRAWL_JOB for the observed_cost field as it progresses.

This example pairs GET_ORGANIZATION_COSTS with the AI's arithmetic reasoning to provide budget guidance. The MCP returns raw usage data; the AI synthesizes it into actionable advice. Costs are estimates only—actual metered charges depend on page size, redirects, and provider-side factors. Always check the crawl job's final observed_cost after completion.

When recurring feeds beat manual checks for pricing intel

A 6-person growth team at a B2B SaaS company tracks three competitors' pricing pages weekly to inform their own packaging decisions. WebCrawlerAPI's Create Feed tool schedules automatic crawls and surfaces only the deltas—no one wastes 20 minutes copy-pasting tables into a spreadsheet. The List Feed Changes tool drops structured JSON into a shared Switchy thread where the PM can flag meaningful shifts without opening tabs. This works cleanly when you monitor under 10 URLs and changes happen weekly or monthly; if you need sub-hour latency or 50+ pages, you'll burn through metered credits fast and should consider a dedicated change-detection service. For quarterly pricing reviews at small-team scale, the feed model is the right call.

Use the one-shot crawl for pre-launch content QA

A 3-person docs team at a developer-tools startup needs to verify that 200 help-center pages render correctly and contain no broken internal links before a product launch. They fire Create Crawl with an items_limit of 250, poll Get Crawl Job until complete, and review the per-page results for 404s or missing metadata in a single Switchy session. The crawl finishes in under 10 minutes and costs a few dollars—cheaper and faster than manually clicking through or setting up Playwright scripts. This approach falls apart if your site requires login or complex JavaScript state; WebCrawlerAPI treats pages as static HTML, so authenticated dashboards or SPAs with client-side routing won't crawl correctly. For public documentation audits, it's the simplest path from 'we should check this' to 'we checked it.'

When feeds surface stale help articles for support teams

A 5-person support team at a fintech company maintains 80 help articles and wants to know when upstream product docs change so they can update customer-facing FAQs. They create a feed pointing at the engineering wiki, set it to crawl weekly, and route List Feed Changes into a Switchy channel where the support lead triages updates every Monday morning. The feed catches renamed endpoints or deprecated features before customers file confused tickets. This setup assumes the source site is public or accepts API-key auth; if your internal wiki lives behind SSO, WebCrawlerAPI can't reach it and you'll need a self-hosted crawler or manual checks. For public or API-accessible knowledge bases under 100 pages, the feed model turns 'someone should watch this' into a 10-minute weekly ritual.