Rate Limits

Learn about Web Scraper API rate limits, job submission speeds, and how domain success rates affect your limits.

Intro

Web Scraper API user account comes with a specific job submission rate limit, tailored to your plan. These limits are designed to accommodate your monthly subscription needs effectively.

Rate limits

The rate at which you can submit jobs depends on your plan size. Refer to the table below for a breakdown of rate limits for regular and enterprise plans:

Plan	Results (maximum)	Total jobs/s	Rendered jobs/s
Free trial	2,000	10	3
Micro	98,000	50	13
Starter	220,000	50	13
Advanced	622,500	50	13
Venture	1,350,000	50	13
Business	3,330,000	100	25
Corporate	8,000,000	100	25
Custom +	Custom	Custom	Custom

Getting rate limit information

Whenever you submit a job to our system, we will send you the rate limit information in the response headers. The header names will be structured like x-ratelimit-limit_name-limit and x-ratelimit-limit_name-remaining. The former denotes the total limit value, while the latter denotes the remaining value. There may be more than one limit applicable to you.

Limiting domain access due to a low success rate

Web Scraper API continuously monitors your success rate on every domain you scrape. If your success rate on any domain goes below 40% (measured over the last 5-minute window), your request rate to that particular domain will be limited to 1 req/s until the success rate recovers.

You may experience a low success rate due to two reasons:

• Web Scraper API's inability to handle the website's bot traffic control systems;
• Your own cookies, headers, or other variables interfering with Web Scraper API's web access logic.

When your requests to a domain are limited due to a low success rate, you will get an API response with the 429 status code and the following message:

{
    "message": "Access to {domain} has been limited to 1 req/s due to a low success rate. If you are using custom headers or cookies, ensure they are correct, then try again. The normal request limit will be restored automatically when the success rate improves."
}

Upon getting this response, please ensure that your request parameters are not the cause for this issue.