Firecrawl

@firecrawl search for 'best practices for API rate limiting' and return the top 3 results with markdown content

Firecrawl performed a web search and scraped the top 3 results. The response includes: (1) A CloudFlare blog post discussing token bucket algorithms and sliding window approaches, (2) A Stripe engineering article on adaptive rate limiting with real-world implementation details, and (3) An AWS architecture guide covering throttling strategies for serverless APIs. Each result contains the page title, URL, and full markdown-formatted content extracted from the page body.

This example demonstrates Firecrawl's search-then-scrape workflow, which combines discovery with content extraction. Be aware that search results depend on external search engine availability, and scraping all top results can consume significant API credits quickly. Each scraped page counts toward your plan's usage limits.

@firecrawl start a crawl from https://docs.example.com with a limit of 50 pages, excluding any URLs matching '/archive/' or '/deprecated/'

Firecrawl initiated crawl job #crawl_abc123 and began discovering pages from the starting URL. The job completed after 8 minutes, successfully crawling 47 pages (3 were excluded by the regex filters). The response includes metadata for each discovered page: URL, title, content in markdown format, and any structured data extracted. Two pages returned 404 errors and were skipped. The full result set is available for download or can be processed page-by-page.

Crawling is Firecrawl's most powerful feature for bulk content extraction, but it's asynchronous and can take minutes to hours depending on site size. The tool polls automatically until completion, which may time out in some contexts. Crawl jobs consume credits per page scraped, so set appropriate limits to avoid unexpected usage.

@firecrawl extract structured data from https://news.ycombinator.com using this schema: for each post, get the title, points, author, and comment count

Firecrawl extracted structured data from the Hacker News homepage and returned a JSON array of 30 posts. Each entry contains: {"title": "Show HN: Open-source tool for...", "points": 247, "author": "username", "commentCount": 83}. The extraction used Firecrawl's LLM-powered parsing to interpret the page structure and map it to the requested schema, handling variations in post layout automatically. The full dataset is ready for analysis or storage.

Structured extraction is ideal when you need consistent data shapes from semi-structured HTML. It works best on pages with repeating patterns (lists, tables, feeds). Note that LLM-based extraction adds latency and cost compared to simple scraping, and accuracy depends on page complexity—always validate critical extractions before relying on them in production workflows.

When Firecrawl beats manual scraping for market intel

A 5-person growth team needs to track competitor pricing pages across 12 SaaS vendors every quarter. Firecrawl's structured extraction tool lets them define a schema once (product name, tier names, monthly price, annual discount) and run it against all 12 URLs in a single job. The crawl completes in under two minutes and returns clean JSON they pipe directly into a Google Sheet. This works because the target pages are public, change infrequently, and follow predictable HTML patterns. If competitors gate pricing behind login walls or use heavy client-side rendering, Firecrawl's scraper will miss data and the team needs a different approach. For quarterly cadence and public pages, this MCP turns a half-day manual task into a 10-minute automated run.

Crawling your own docs to feed a support bot

A 3-person support team maintains 200+ help articles across Notion and a legacy WordPress site. They want to index all content into a vector database so their AI assistant can answer customer questions with current links. Firecrawl's crawl job starts from the root URL, discovers all subpages via sitemap, and extracts markdown or plain text in one pass. The team runs this weekly via a cron job and uploads the output to Pinecone. This setup works because the docs are public, the sitemap is accurate, and 200 pages fit comfortably under Firecrawl's rate limits. If the knowledge base grows past 2,000 pages or requires authentication, the crawl becomes slow and the team should evaluate a dedicated docs-indexing service. For small-to-mid-size public doc sites, Firecrawl is the fastest path to a refreshed support context.

When Firecrawl's search tool replaces manual prospecting

A 2-person sales team at a niche B2B startup needs to build a list of 50 target accounts from a public industry directory. They use Firecrawl's search tool with a query like 'manufacturing companies ISO certified California' and scrape the top 20 results. The structured extraction pulls company name, location, and contact page URL into a CSV they import to HubSpot. This works because the directory pages are public, the search query is specific enough to surface relevant results, and 20 pages stay within a single API call. If the directory requires login, uses aggressive bot detection, or the team needs 500+ leads, Firecrawl will hit rate limits or return incomplete data. For one-time or monthly prospecting runs under 50 targets, this MCP turns a 4-hour manual task into a 15-minute script.