Documentation has been updated: see help center and changelog in one place.

Rate Limits

Intro

Web Scraper API user account comes with a specific job submission rate limit, tailored to your plan. These limits are designed to accommodate your monthly subscription needs effectively.

Rate limits

The rate at which you can submit jobs depends on your plan size. Refer to the table below for a breakdown of rate limits for regular and enterprise plans:

Plan
Results
Total jobs/s
Rendered jobs/s

Free trial

5,000

10

3

Micro

36,296

50

13

Starter

76,154

50

13

Advanced

199,200

50

13

Venture

433,913

50

13

Business

999,000

100

25

Corporate

2,105,263

100

25

Custom +

Custom

Custom

Custom

Getting rate limit information

Whenever you submit a job to our system, we will send you the rate limit information in the response headers. The header names will be structured like x-ratelimit-limit_name-limit and x-ratelimit-limit_name-remaining. The former denotes the total limit value, while the latter denotes the remaining value. There may be more than one limit applicable to you.

Limiting domain access due to a low success rate

Web Scraper API continuously monitors your success rate on every domain you scrape. If your success rate on any domain goes below 40% (measured over the last 5-minute window), your request rate to that particular domain will be limited to 1 req/s until the success rate recovers.

You may experience a low success rate due to two reasons:

  • Web Scraper API's inability to overcome the website's anti-bot measures;

  • Your own cookies, headers, or other variables interfering with Web Scraper API's unblocking logic.

When your requests to a domain are limited due to a low success rate, you will get an API response with the 429 status code and the following message:

{
    "message": "Access to {domain} has been limited to 1 req/s due to a low success rate. If you are using custom headers or cookies, ensure they are correct, then try again. The normal request limit will be restored automatically when the success rate improves."
}

Upon getting this response, please ensure that your request parameters are not the cause for this issue.

Last updated

Was this helpful?