Centralized Management: View all active and inactive crawlers in one place
Quick Status Overview: Monitor job health, costs, and performance at a glance
Intuitive Navigation: Easy access to create, configure, and manage crawlers
Real-time Progress Tracking: Watch your crawlers work with live updates
Interactive Job Tables: Search, and filter through crawler job list
Detailed insights: Cost tracking and success monitoring
Step-by-Step Configuration: Guided process for creating new crawlers
Download job results directly from the dashboard.
Smarter, job-based crawling engine
Starts from a single URL and automatically discovers and scrapes new pages based on your rules.
Skips duplicate URLs to avoid loops and wasted credits.
Handles failed requests gracefully (failures don’t cost credits, and full reasons are included in job summaries).
Automatically stops at your credit budget or depth limit, so you stay in control of cost and scope.
Streams page results in real time to your webhook, plus delivers a full job summary when the crawl is done.
Flexible scheduling: You can run a crawl once or schedule it hourly, daily, weekly, or monthly, all while keeping budget and depth controls in place.
Whether you’re crawling 10 pages or 10,000, Crawler 2.0 runs each job from start to finish and sends every page result to your webhook as it’s processed.
Last updated