Scheduler
Last updated
Last updated
Scheduler is a free feature of the Scraper APIs that lets you automate recurring scraping and parsing jobs by creating schedules.
Check out the video tutorial below to learn more about Scheduler and how it works.
We advise using Scheduler together with the Upload to Cloud Storage feature. This way, you can set up your schedule and receive regular data updates in your storage without trying to fetch results from our system.
IMPORTANT: Scheduler is a powerful tool that can quickly raise your service bill. We advise testing it with a few job items and a limited number of repeats to ensure you get the correct data at the right intervals. Once that's established, you can stop the test schedule and create a new, scaled-up schedule.
When creating a new schedule, follow the simple steps below.
Tell us how often we should repeat the jobs by submitting a cron schedule expression;
Give us a bunch of job parameter sets that we should execute at scheduled times;
Let us know when to stop by submitting an end time.
See here to find a code example for submitting a new schedule.
HINT: Find out how to form job parameter sets by visiting the documentation of the particular Scraper API (SERP / E-Commerce / Web) you want to use.
NOTE: You can also download and import this Postman collection to try out all our Scheduler endpoints. New to Postman? Learn more about this tool here.
Scheduler has several endpoints you can use to control the service:
Use this endpoint to initiate a new schedule.
Endpoint: https://data.oxylabs.io/v1/schedules
Method: POST
Authentication: Basic
Request headers: Content-Type: application/json
Input
- required parameter
The payload below will make Scheduler run two jobs schedule at 03:00 on Mondays until end_time
(inclusive).
The response below confirms that the schedule was created successfully.
Use this endpoint to get the list of all schedules associated with your user account.
Endpoint: https://data.oxylabs.io/v1/schedules
Method: GET
Authentication: Basic
This endpoint returns the list of all schedule IDs associated with the user account making the request.
See the sample response below.
Use this endpoint to get information about a specific schedule.
Endpoint: https://data.oxylabs.io/v1/schedules/{id}
Method: GET
Authentication: Basic
The payload below contains a sample schedule info response.
NOTE: Currently,items
(i.e., the jobs to be executed as part of a schedule) are missing in the output. We are aware of this and will add them shortly.
Use this endpoint to activate or deactivate a particular schedule.
Endpoint: https://data.oxylabs.io/v1/schedules/{id}/state
Method: PUT
Authentication: Basic
Use this endpoint to stop or restart a schedule.
By setting active
to false
, you can stop the execution of a particular schedule.
If you set active
to true
, you can reactivate a previously stopped schedule.
Output
The standard response is an empty response body with a 202
status code.
For API response codes, refer to API section.
Parameter | Description | Default Value |
---|---|---|
Key | Description | Type |
---|---|---|