Crawler General Availability Update

We’re excited to announce that Crawler v2.0 has officially moved out of beta

Crawler v2.0 is officially out of beta and generally available to all users. This release introduces plan-based features that unlock a more powerful crawling experience for paid users, while keeping the core functionality accessible on the Free plan.

What's Changed

Link Depth

• Free plan: link depth is now capped at 1

• Paid plan: link depth is uncapped — configure as deep as your use case requires

• If a Free plan user submits a job with link_depth > 1, the dashboard shows an inline validation message and the API returns a 403 error

Scheduling

• Free plan: scheduled Crawler jobs are no longer supported

• Paid plan: full scheduling access with interval-based configurations

---

Impact on Existing Jobs

Free plan users with active scheduled jobs or jobs with link_depth > 1 created during the beta are not immediately affected. These jobs will continue running through the grace period ending May 15, 2025, after which they will be paused automatically.

Recommendations for affected users on Free plans:

• Review your active Crawler jobs before May 15

• Upgrade to a paid plan to keep them running without interruption

• If you choose to stay on the Free plan, reconfigure affected jobs to link_depth = 1 and remove scheduling before the grace period ends

---

No action required for paid plan users. All existing and new jobs continue running without any restrictions.

---

Questions? Contact our support team.

VISIT CRAWLER DOCS