Push-Pull: Batch

Push-Pull is our recommended integration method for reliably handling large amounts of data.

Visit the Oxylabs GitHub repository for a complete working example of Push-Pull integration in Python.

Push-Pull is an asynchronous integration method. Upon job submission, you will promptly receive a JSON response containing all job details, including job parameters, ID, and URLs for result download and status checking. Once your job is processed, we will update you via a JSON payload sent to your server, if you provided a callback URL. Results remain available for retrieval for at least 24 hours after completion.

With Push-Pull, you can upload your results directly to your cloud storage (AWS S3 or Google Cloud Storage).

If you prefer not to set up a service for incoming callback notifications, you can simply retrieve your results periodically (polling).

You can also explore how Push-Pull works using Postman.

Batch Query

Scraper APIs supports submitting up to 5,000 query or url parameter values within a single batch request.

Endpoint

POST https://data.oxylabs.io/v1/queries/batch

The system will handle every query or url submitted as a separate job. If you provide a callback URL, you will get a separate call for each keyword. Otherwise, our initial response will contain job ids for all keywords. For example, if you sent 50 keywords, we will return 50 unique job ids.

IMPORTANT: With /batch endpoint, you can only submit lists of queryor urlparameter values (depending on the source you use). All other parameters should have singular values.

Input

You need to post query parameters as a JSON payload. Here is how you submit a batch job:

curl --user "user:pass1" \
'https://data.oxylabs.io/v1/queries/batch' \
-H 'Content-Type: application/json' \
-d '@keywords.json'

You may notice that the code example above doesn't explain how the JSON payload should be formatted and points out to a pre-made JSON file. Below is the content of keywords.json file, containing multiple query parameter values:

{  
   "query":[  
      "adidas",
      "nike",
      "reebok"
   ],
   "source": "google_shopping_search",
   "domain": "com",
   "callback_url": "https://your.callback.url"
}

...and here is a keywords.json batch input file, containing multiple URLs:

{  
   "url":[  
      "https://example.com/url1.html",
      "https://example.com/url2.html",
      "https://example.com/url3.html"
   ],
   "source": "universal",
   "callback_url": "https://your.callback.url"
}

Output

The API will respond with a JSON object, containing the job information for each job created. The response will be similar to this:

{
  "queries": [
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678900987654321",
      {...}
      "query": "adidas",
      "source": "google_shopping_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "12345678901234567890",
      {...}
      "query": "nike",
      "source": "google_shopping_search",
      {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/12345678901234567890/results",
          "method": "GET"
        }
      ]
    },
    {
      "callback_url": "https://your.callback.url",
      {...}
      "created_at": "2019-10-01 00:00:01",
      "domain": "com",
      "id": "01234567899876543210",
	  {...}
      "query": "reebok",
      "source": "google_shopping_search",
	  {...}
          "rel": "results",
          "href": "http://data.oxylabs.io/v1/queries/01234567899876543210/results",
          "method": "GET"
        }
      ]
    }
  ]
}

Data dictionary

For detailed descriptions of the job input parameters, please consult the table below or refer to the specific documentation pages for the scrapers you are interested in.

KeyDescriptionType

created_at

The datetime the job was created at.

String

client_id

The numerical ID associated with the username of the client making the request.

String

client_notes

Submitted notes by the client when sending a job.

String

content_encoding

Add this parameter if you are downloading images. Learn more here.

String

id

The unique ID of the job.

String

statuses

The status code of the scraping or parsing job. You can see the status codes described here.

Integer

status

The status of the job. pending means the job is still being processed. done means we've completed the job. faulted means we came across errors while trying to complete the job and gave up at it.

String

subdomain

The subdomain of the website.

String

updated_at

The datetime the job was last updated at. For jobs that are finished (status is done or faulted), this datetime indicates when the job was finished.

String

links

The list of links, related to the provided input.

JSON Array

links:rel

The link type. self URL contains the metadata of the job, while results URL contains the job results.

String

links:href

The URL to the resource.

String

links:method

The HTTP method that should be used to interact with a given URL.

String

Callback

The callback is a POST request we send to your machine, informing that the data extraction task is completed and providing a URL to download scraped content. This means that you no don't need to check job status manually. Once the data is here, we will let you know, and all you need to do now is to retrieve it.

Input

# This is a simple Sanic web server with a route listening for callbacks on localhost:8080.
# It will print job results to stdout.
import requests
from pprint import pprint
from sanic import Sanic, response


AUTH_TUPLE = ('user', 'pass1')

app = Sanic()


# Define /job_listener endpoint that accepts POST requests.
@app.route('/job_listener', methods=['POST'])
async def job_listener(request):
    try:
        res = request.json
        links = res.get('_links', [])
        for link in links:
            if link['rel'] == 'results':
                # Sanic is async, but requests are synchronous, to fully take
                # advantage of Sanic, use aiohttp.
                res_response = requests.request(
                    method='GET',
                    url=link['href'],
                    auth=AUTH_TUPLE,
                )
                pprint(res_response.json())
                break
    except Exception as e:
        print("Listener exception: {}".format(e))
    return response.json(status=200, body={'status': 'ok'})


if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8080)

Output

{  
   "created_at":"2019-10-01 00:00:01",
   "updated_at":"2019-10-01 00:00:15",
   "locale":null,
   "client_id":163,
   "user_agent_type":"desktop",
   "source":"google_shopping_search",
   "pages":1,
   "subdomain":"www",
   "status":"done",
   "start_page":1,
   "parse":0,
   "render":null,
   "priority":0,
   "ttl":0,
   "origin":"api",
   "persist":true,
   "id":"12345678900987654321",
   "callback_url":"http://your.callback.url/",
   "query":"adidas",
   "domain":"com",
   "limit":10,
   "geo_location":null,
   {...}
   "_links":[
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321",
         "method":"GET",
         "rel":"self"
      },
      {  
         "href":"https://data.oxylabs.io/v1/queries/12345678900987654321/results",
         "method":"GET",
         "rel":"results"
      }
   ],
}

Check Job Status

If you provided a valid callback URL when submitting your job, we will notify you upon completion by sending a JSON payload to the specified callback URL. This payload will indicate that the job has been completed and its status set to done.

However, if you submitted a job without using callback service, you can check the job status manually. Retrieve the URL from the href field in the rel:self section of the response message received after job submission. The URL for checking the job status will resemble the following: http://data.oxylabs.io/v1/queries/12345678900987654321. Querying this URL will return the job information, including its current status.

Endpoint

GET https://data.oxylabs.io/v1/queries/{id}

Input

curl --user "user:pass1" \
'http://data.oxylabs.io/v1/queries/12345678900987654321'