Documentation has been updated: see help center and changelog in one place.
⭐Explore
LogoLogo
Oxylabs dashboardContact usProduct
English
  • Documentation
  • Help center
  • Changelog
  • Overview
  • PROXIES
    • Integration Guides
      • Get IP Address for Integrations
      • Residential Proxies Guides
        • AdsPower
        • Android
        • ClonBrowser
        • Dolphin Anty
        • FoxyProxy
        • Ghost Browser
        • GoLogin
        • Helium Scraper
        • Incogniton
        • iOS
        • Kameleo
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Multilogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • ParseHub
        • Playwright
        • Proxifier
        • Puppeteer
        • Selenium
        • SEO Neo
        • SessionBox
        • Shadowrocket
        • Super Proxy
        • SwitchyOmega
        • Ubuntu
        • VMLogin
        • WebHarvy
        • Hidemyacc
      • ISP Proxies Guides
        • AdsPower
        • Android
        • Dolphin Anty
        • FoxyProxy
        • GoLogin
        • Incogniton
        • iOS
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Multilogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • Proxifier
        • SEO Neo
        • Shadowrocket
        • Sphere
        • Super Proxy
        • SwitchyOmega
        • Ubuntu
        • Hidemyacc
      • Mobile Proxies Guides
        • AdsPower
        • Android
        • ClonBrowser
        • Dolphin Anty
        • Ghost Browser
        • GoLogin
        • Helium Scraper
        • Incogniton
        • iOS
        • Kameleo
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Multilogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • ParseHub
        • Playwright
        • Proxifier
        • Puppeteer
        • Selenium
        • SEO Neo
        • SessionBox
        • Shadowrocket
        • SwitchyOmega
        • Ubuntu
        • VMLogin
        • WebHarvy
      • Dedicated Datacenter Proxies Guides
        • Enterprise
          • Dolphin Anty
          • FoxyProxy
          • GoLogin
          • Lalicat Browser
          • MoreLogin
          • MuLogin
          • Nstbrowser
          • Octoparse
          • Oxy® Proxy Extension for Chrome
          • Proxifier
          • SEO Neo
          • Shadowrocket
          • Sphere
          • Super Proxy
          • SwitchyOmega
          • Ubuntu
          • Hidemyacc
        • Self-Service
          • Android
          • Dolphin Anty
          • FoxyProxy
          • GoLogin
          • iOS
          • Lalicat Browser
          • MacOS
          • MoreLogin
          • MuLogin
          • Nstbrowser
          • Octoparse
          • Oxy® Proxy Extension for Chrome
          • Proxifier
          • SEO Neo
          • Shadowrocket
          • Sphere
          • Super Proxy
          • SwitchyOmega
          • Ubuntu
          • Hidemyacc
      • Datacenter Proxies Guides
        • AdsPower
        • Android
        • Dolphin Anty
        • FoxyProxy
        • GoLogin
        • iOS
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • Proxifier
        • SEO Neo
        • Shadowrocket
        • Super Proxy
        • SwitchyOmega
        • Ubuntu
        • Hidemyacc
    • Residential Proxies
      • Getting Started
      • Making Requests
        • Entry Node for China
      • Location Settings
        • Country
        • City
        • State
        • Continent
        • ZIP/Postal code
        • Coordinates
        • ASN Targeting
      • Session Control
        • Sticky Proxy Entry Nodes
      • Protocols
      • Whitelisting IPs
        • Requests with Whitelisted IPs
      • Endpoint Generator
      • Restricted Targets
      • Public API
      • Response Codes
    • ISP Proxies
      • Making Requests
      • Proxy List
      • Proxy Rotation
      • Location Settings
      • Protocols
      • Whitelisting IPs
      • Response Codes
      • Restricted Targets
      • Fair usage policy
    • Mobile Proxies
      • Getting Started
      • Making Requests
        • Entry Node for China
      • Location Settings
        • Country
        • City
        • State
        • Continent
        • Coordinates
        • ASN Targeting
      • Session Control
        • Sticky Proxy Entry Nodes
      • Protocols
      • Whitelisting IPs
      • Endpoint Generator
      • Restricted Targets
      • Public API
      • Response Codes
    • Datacenter Proxies
      • Proxy List
      • IP Control
      • Select Country
      • Protocols
      • Whitelisting
      • Response Codes
      • Restricted Targets
      • Fair usage policy
      • Free Datacenter IPs
    • Dedicated Datacenter Proxies
      • Enterprise
        • Getting Started
        • Proxy List
        • Making Requests
        • Protocols
        • Whitelisting IPs
          • Dashboard
          • RESTful
            • Getting Whitelisted IPs List
            • Adding a Whitelisted IP
            • Removing a Whitelisted IP
            • Saving Changes (5min Cooldown)
        • Datacenter Proxy API
        • Proxy Rotator - Optional
        • Response Codes
      • Self-Service
        • Getting Started
        • Making Requests
        • Proxy List
        • Proxy Rotation
        • Location Settings
        • Protocols
        • Whitelisting IPs
        • Response Codes
        • Restricted Targets
        • Fair usage policy
    • Dedicated ISP Proxies
      • Getting Started
      • Proxy List
      • Making Requests
      • Protocols
      • Whitelisting IPs (RESTful)
        • Getting Whitelisted IPs List
        • Adding a Whitelisted IP
        • Removing a Whitelisted IP
        • Saving Changes (5min Cooldown)
      • Proxy API
      • Proxy Rotator - Optional
      • Response Codes
  • Advanced proxy solutions
    • Web Unblocker
      • Getting Started
      • Making Requests
        • Session
        • Geo-location
        • Headers & Cookies
        • Custom status code
        • POST requests
      • Headless Browser
        • JavaScript rendering
        • Browser instructions (Beta)
          • List of instructions
      • Sample Response
      • Response Codes
      • Rate Limits
      • Migration Guides
        • From Bright Data Web Unlocker
      • Usage Statistics
      • Billing Information
  • VIDEO DATA
    • High-Bandwidth Proxies
      • YouTube Downloader (yt_dlp) integration
  • Video Data API
  • Scraping Solutions
    • Web Scraper API
      • Integration Methods
        • Realtime
        • Push-Pull
        • Proxy Endpoint
      • Features
        • Localization
          • Proxy Location
          • SERP Localization
          • E-Commerce Localization
          • Domain, Locale, Results Language
        • JS Rendering & Browser Control
          • JavaScript Rendering
          • Browser Instructions
            • List of instructions
          • Capturing network requests (Fetch/XHR)
        • Result Processing & Storage
          • Dedicated Parsers
          • Custom Parser
            • Getting started
            • Parsing instruction examples
            • List of functions
              • Function examples
          • Download Images
          • Cloud Storage
        • HTTP Context & Job Management
          • Headers, Cookies, Method
          • User Agent Type
          • Client Notes
        • Scheduler
      • Solutions for AI Workflows
        • Model Context Protocol (MCP)
        • LangChain
        • LlamaIndex
      • Targets
        • Google
          • Search
            • Web Search
            • AI Overviews
            • Image Search
            • News Search
            • Local Search
            • Reverse Image Search
            • Google Suggest
          • Ads Max
          • Shopping
            • Shopping Product
            • Shopping Search
            • Shopping Pricing
          • Trends: Explore
          • Travel: Hotels
          • Lens
          • URL
        • Amazon
          • Product
          • Search
          • Pricing
          • Sellers
          • Best Sellers
          • Reviews
          • Questions & Answers
          • URL
        • YouTube
          • YouTube Scraping Guide for AI
          • YouTube Search
          • YouTube Video Trainability
          • YouTube Metadata
          • YouTube Downloader
          • YouTube Transcript
        • Generic Target
        • Walmart
          • Search
          • Product
        • Ebay
        • Etsy
          • Search
          • Product
        • Bing
          • Search
          • URL
        • ChatGPT
        • North American E-Commerce
          • Best Buy
            • Search
            • Product
          • Kroger
            • Product
            • Search
            • URL
          • Lowe's
            • Search
            • Product
            • URL
          • Target
            • Search
            • Product
            • Category
          • Bed Bath & Beyond
          • Costco
          • Menards
          • Petco
          • Staples
          • Grainger
          • Instacart
        • European E-Commerce
          • Allegro
            • Search
            • Product
          • Idealo
          • Mediamarkt
          • Cdiscount
        • Asian E-Commerce
          • Alibaba
          • Aliexpress
          • Lazada
          • Rakuten
          • Tokopedia
          • Flipkart
          • Avnet
          • Indiamart
        • Latin American E-Commerce
          • Mercado Livre
          • Magazine Luiza
          • Falabella
          • Dcard
      • Restricted Targets
      • Response Codes
      • Usage and Billing
        • Usage Statistics
        • Traffic and Billing
        • Rate Limits
    • OxyCopilot
    • Unblocking Browser
      • Chrome
      • Firefox
      • Restricted Targets
      • Integration with MCP
      • Troubleshooting Guide
  • Dashboard
    • Teams
    • Billing Information
      • Accessing Billing Information
      • Managing Payment Methods
      • Updating Billing Information
      • Canceling a Subscription
    • IP Replacement
  • Guides for Scraper APIs
    • Python SDK
    • Go SDK
    • Forming Requests
    • Forming URLs
    • Using Postman
  • Useful links
    • Oxylabs Dashboard
    • Release Notes
    • Network status
    • Open Source Tools
      • Oxy Parser
      • Oxy Mouse
      • Web Scraper API Scheduler
    • Discord Community
    • GitHub
    • Scraping Experts
  • SUPPORT
    • FAQ
    • Have a Question?
Powered by GitBook
On this page
  • Quick Start
  • Endpoints
  • Create a new schedule
  • Get all schedules
  • Get runs information
  • Get scheduled jobs
  • Get schedule information
  • Deactivate or reactivate a schedule
  • API response codes

Was this helpful?

  1. Scraping Solutions
  2. Web Scraper API
  3. Features

Scheduler

PreviousClient NotesNextSolutions for AI Workflows

Last updated 3 months ago

Was this helpful?

Scheduler is a free feature of the Scraper APIs that lets you automate recurring scraping and parsing jobs by creating schedules.

Check out the video tutorial below to learn more about Scheduler and how it works.

We advise using Scheduler together with the Upload to Cloud Storage feature. This way, you can set up your schedule and receive regular data updates in your storage without trying to fetch results from our system.

IMPORTANT: Scheduler is a powerful tool that can quickly raise your service bill. We advise testing it with a few job items and a limited number of repeats to ensure you get the correct data at the right intervals. Once that's established, you can stop the test schedule and create a new, scaled-up schedule.

Quick Start

When creating a new schedule, follow the simple steps below.

  1. Tell us how often we should repeat the jobs by submitting a cron schedule expression;

  2. Give us a bunch of job parameter sets that we should execute at scheduled times;

  3. Let us know when to stop by submitting an end time.

See here to find a code example for submitting a new schedule.

NOTE: You can also download and import this Postman collection to try out all our Scheduler endpoints. New to Postman? Learn more about this tool here.

Endpoints

Scheduler has several endpoints you can use to control the service:

  • Create a new schedule

  • Get all schedules

  • Get runs information

  • Get scheduled jobs

  • Get schedule information

  • Deactivate or reactivate a schedule

Create a new schedule

Overview

Use this endpoint to initiate a new schedule.

  • Endpoint: https://data.oxylabs.io/v1/schedules

  • Method: POST

  • Authentication: Basic

  • Request headers: Content-Type: application/json

Input

Parameter
Description
Default Value

cron

-

items

List of Scraper APIs job parameter sets that should be executed as part of the schedule.

-

end_time

The time at which the schedule should stop running. NB: the end time is inclusive.

-

- required parameter

NOTE: For guidance on putting together job parameter sets for the items part of your Scheduler payload, refer to the documentation page of the particular scraper you would like to use (e.g. Google, Amazon, etc.).

The payload below will make Scheduler run two jobs schedule at 03:00 on Mondays until end_time (inclusive).

{
  "cron": "0 3 * * 1",
  "items": [
    {"source": "universal", "url": "https://ip.oxylabs.io"},
    {"source": "google_search", "query": "stuff"}
  ],
  "end_time": "2032-12-21 12:34:45"
}

Output

The response below confirms that the schedule was created successfully.

{
    "schedule_id": 4134906379157007223,
    "active": true,
    "items_count": 3,
    "cron": "0 3 * * 1",
    "end_time": "2032-12-21 12:34:45",
    "next_run_at": "2022-06-06 10:15:00"
}

Get all schedules

Overview

Use this endpoint to get the list of all schedules associated with your user account.

  • Endpoint: https://data.oxylabs.io/v1/schedules

  • Method: GET

  • Authentication: Basic

Output

This endpoint returns the list of all schedule IDs associated with the user account making the request.

See the sample response below.

{
    "schedules": [
        1764178033254455101,
        2885262175311057587,
        3251365810325795747,
        4134906379157007223,
        4164931482277157062
    ]
}

Get runs information

Overview

Use this endpoint to get information about a list of all runs in a schedule with the metadata of each job and each run’s success rate.

  • Endpoint: https://data.oxylabs.io/v1/schedules/{id}/runs

  • Method: GET

  • Authentication: Basic

Output

The payload below contains a sample /runs endpoint response.

{
 "runs": [
        {
            "run_id": 25037485,
            "jobs": [
                {
                    "id": 7300439540206948353,
                    "create_status_code": 202,
                    "result_status": "done",
                    "created_at": "2025-02-26 09:00:21",
                    "result_created_at": "2025-02-26 09:00:23"
                },
                {
                    "id": 7300439540169188353,
                    "create_status_code": 202,
                    "result_status": "done",
                    "created_at": "2025-02-26 09:00:21",
                    "result_created_at": "2025-02-26 09:00:22"
                },
                {
                    "id": 7300439540198551553,
                    "create_status_code": 202,
                    "result_status": "done",
                    "created_at": "2025-02-26 09:00:21",
                    "result_created_at": "2025-02-26 09:00:23"
                }
            ],
            "success_rate": 1
        }
}
Key
Description
Type

runs

A collection of run objects that represent execution instances of a scheduled task or workflow.

Array

runs:run_id

A unique identifier for the specific run instance.

Integer

runs:jobs

A collection of job objects that were executed as part of this run.

Array

runs:success_rate

The ratio of successful jobs to total jobs in this run (ranges from 0 to 1).

Number

runs:jobs:id

A unique Oxylabs identifier for the specific job.

Integer

runs:jobs:create_status_code

HTTP status code returned when the job was created, indicating the initial acceptance of the job request.

Integer

runs:jobs:result_status

The execution status of the job (e.g., "done", "failed", "pending").

String

runs:jobs:created_at

Timestamp when the job was created

String

runs:jobs:result_created_at

Timestamp when the job completed and produced a result

String

Get scheduled jobs

Overview

Use this endpoint to get the list of scraping jobs executed as a result of running a schedule.

  • Endpoint: https://data.oxylabs.io/v1/schedules/{id}/jobs

  • Method: GET

  • Authentication: Basic

Output

The payload below contains a sample schedule info response.

{
    "jobs": [
        7300439540206948353,
        7300439540169188353,
        7300439540198551553,
        ...,
      ]
}

Get schedule information

Overview

Use this endpoint to get information about a specific schedule.

  • Endpoint: https://data.oxylabs.io/v1/schedules/{id}

  • Method: GET

  • Authentication: Basic

Output

The payload below contains a sample schedule info response.

{
    "schedule_id": 1764178033254455101,
    "active": true,
    "items_count": 3,
    "cron": "0 3 * * 1",
    "end_time": "2032-12-21 12:34:45",
    "next_run_at": "2022-06-06 10:18:00",
    "links": [
        {
            "rel": "runs",
            "href": "/v1/schedules/1764178033254455101/runs",
            "method": "GET"
        },
        {
            "rel": "jobs",
            "href": "/v1/schedules/1764178033254455101/jobs",
            "method": "GET"
        }
    ],
    "stats": {
        "total_job_count": 3,
        "job_create_outcomes": [
            {
                "status_code": 202,
                "job_count": 3,
                "ratio": 1
            }
        ],
        "job_result_outcomes": [
            {
                "status": "done",
                "job_count": 2,
                "ratio": 0.67
            },
            {
                "status": "faulted",
                "job_count": 1,
                "ratio": 0.33
            }
        ]
    }
}

Key
Description
Type

schedule_id

The unique ID of the schedule.

Integer

active

Is the schedule active right now?

Boolean

items_count

The number of items (jobs) in the schedule.

Integer

cron

The cron expression associated with the schedule.

String

end_time

The time upon which the schedule will stop being repeated.

String

next_run_at

The time upon which the schedule will run next.

String

links

A collection of link objects that define available API endpoints related to a schedule resource.

Array

links:rel

The relationship identifier that explains the purpose of the link relative to the parent resource.

String

links:href

The URL path to the API endpoint. Represents the resource location that can be accessed.

String

links:method

The HTTP method to be used when accessing this endpoint.

String

stats

Contains stats job creation and job completion statistics.

JSON Object

stats:total_job_count

The number of items (jobs) in the schedule.

Integer

stats:job_create_outcomes

Contains job creation statistics.

JSON Array

stats:job_create_outcomes:status_code

The status code received in response to an attempt to execute the schedule (create a scraping/parsing job).

Integer

stats:job_create_outcomes:job_count

The number of job creation attempts that resulted in that particular status code.

Integer

stats:job_create_outcomes:ratio

The ratio between the number of job creation attempts that resulted in that particular attempt and the total number of job creation attempts.

Float

job_result_outcomes

Contains the outcome statistics of scraping/parsing jobs executed as part of the schedule.

JSON Array

status

The job status. Possible values: pending (the job is still being processed), done (the job has been completed successfully), faulted (the job has failed).

String

job_count

The number of jobs that resulted in that particular status.

Integer

ratio

The ratio between the number of jobs with that particular status and the total number of jobs created.

Float

Deactivate or reactivate a schedule

Overview

Use this endpoint to activate or deactivate a particular schedule.

  • Endpoint: https://data.oxylabs.io/v1/schedules/{id}/state

  • Method: PUT

  • Authentication: Basic

Input

Use this endpoint to stop or restart a schedule.

By setting active to false, you can stop the execution of a particular schedule.

If you set active to true, you can reactivate a previously stopped schedule.

{
  "active": false
}

Output

null

The standard response is an empty response body with a 202 status code.

API response codes

For API response codes, refer to API section.

Cron schedule expression. It determines how often the submitted schedule will run. Read more and .

here
here
Step-by-step guide on automating your recurring scraping jobs using Scheduler