AI tools for “crawl”

Scrape, clean, match: ready data APIs and AI scrapers that end in a competitor price comparison. Pay as you go, 5 free requests. REST API, CSV, MCP.

Web infrastructure for AI to search, extract, monitor, and reason over the world's information. Introducing Parallel Decisions API: the first decision engine with world knowledge built in. Learn more.[Learn…

Every public LLM benchmark we can scrape, folded into one daily ranking with uncertainty, plus saturation forecasts per benchmark and a changelog of what moved.

SummaryTube is the best AI YouTube video summarizer. Get instant AI summaries of any YouTube video for free — extract key insights, moments & transcripts in seconds.

Deep research with Firecrawl. Investigate questions needing multi-source synthesis and return a cited answer, report or table. Use for deep dives, cross-source comparisons, market and company investigations, and…

…papers, or library and API documentation. When using Firecrawl for the request, use firecrawl_search to find sources and firecrawl_scrape to read a supplied or discovered URL. Firecrawl also collects structured data.…

Free PDF splitter to extract pages, split by page range, or save every page as a separate PDF. 100% in your browser — files never leave your device.

…existing website through a four-phase pipeline — extract (crawl and capture the current site), direct (set a visual direction), prototype (generate redesigned HTML), and migrate (emit a deployable static site). Tracks…

…by orchestrating the prep cascade — a full-inventory crawl (extract --prep), page-type and module-catalog confirmation (direct --prep), archetype prototypes plus design canon (prototype --prep), and asset…

Crawl an existing website (capped, multi-page) and seed stardust/current/ with PRODUCT.md, DESIGN.md, DESIGN.json, a per-page inventory, and the consolidated brand surface — the captured design system, palette,…

Crawl and validate all internal and external links across an AEM Edge Delivery Services site. Uses the query index or sitemap to discover pages, extracts links from .plain.html renditions, checks HTTP status codes, and…

…an asset, create a data product, run the data source crawl, delete a catalog asset, attach a conventions document to a table as a catalog skill asset, which assets in my project are missing descriptions or owners…

Curate Gaia bot crawler branches into real-skill catalog entries and clean up processed bot/* branches. Use when the user says: "process bot branches", "curate the crawl output", "clean up bot proposals", "run bot…

Firecrawl gives AI agents and apps fast, reliable web context with strong search, scraping, and interaction tools. One install command sets up three skill segments: live CLI tools, app-integration build skills, and…

…REFUSES TO RUN without an explicit gbrain.yml `archive-crawler.scan_paths:`…

Find web sources with query-relevant page excerpts and optional full-page content, and discover workflows, data APIs, and indexes. Use for web research or finding structured records, listings, transcripts, and datasets.…

Read a known webpage or execute a discovered workflow or data-provider capability. Use for page content or structured results once the URL or tool is selected.

Diagnose whether a website's content is visible to AI crawlers (ClaudeBot, GPTBot, PerplexityBot, Bytespider, Meta-ExternalAgent) that do NOT execute JavaScript, and prescribe framework-specific fixes. Use this skill…

Any live-web task via the Firecrawl CLI — including ordinary web research: searching the web, reading or extracting pages, gathering sources, discovering site URLs, bulk extraction, downloading a site, change alerts, or…

Search, scrape, crawl, extract structured data, and monitor web pages via the ScrapeGraph AI CLI. Use when the user asks to search the web, scrape a webpage, grab content from a URL, extract JSON from a site, crawl…