DataPipeline Update: 10× Larger Jobs & 30-Day Data Retention Policy

Two significant updates to DataPipeline are shipping together today.

1. 10x more URLs per job — up to 100,000

You can now submit up to 100,000 URLs, ASINs, or keywords in a single DataPipeline project, up from the previous 10,000 limit. Larger jobs run in a single submission with no manual splitting required.

Just create a new project and start with a bigger list.

2. Result retention policy — 30 days

DataPipeline job results will now be automatically deleted 30 days after job completion. This applies to all job output files and result payloads. Every job now includes a results_expire_at field in the API response so you always know exactly when your results will be removed.

Existing projects — grace period

If you have existing DataPipeline projects created before this release, your results are safe until 2026-05-13. Download any existing data before that date. After that, the standard 30-day policy applies to all jobs going forward.

Why both changes together?

The 100,000 URL limit significantly increases the result payload sizes we store per job. Introducing a clear retention policy at the same time keeps infrastructure costs sustainable as job volume scales — and gives you a predictable, well-communicated window to retrieve your data.

VISIT DATAPIPELINE DOCS