Forming Requests

Read detailed guides on how to get started and make requests using each of our Scraper APIs.

SERP Scraper API

Getting started

Select the search engine you want to scrape: Google, Bing, Other Search Engines.

Request sample

curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "google_search",
        "query": "adidas"
    }'

We use synchronous Realtime integration method in our examples. If you would like to use Proxy Endpoint or asynchronous Push-Pull integration, refer to the integration methods section.

Forming a request

  1. Pick your integration method: synchronous (Realtime, Proxy Endpoint) or asynchronous (Push-Pull).

  2. When forming a request, include the following elements:

  1. Endpoint:

POST https://realtime.oxylabs.io/v1/queries
  1. Username and password (HTTP authentication). Create API user credentials either during your trial sign-up or product purchase.

USERNAME:PASSWORD

If you need more than one API user for your account, please contact our customer support or message our 24/7 live chat support.

  1. Content-type. When submitting jobs, always add this header:

Content-Type: application/json
  1. Payload:

    1. source - This parameter sets the scraper that will be used to process your request.

    2. URL or query - Provide the URL or query for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.

    3. Additional parameters: Optionally, you can include additional parameters such as geo_location, user_agent_type, parse, render and more to customize your scraping request.

{
    "source": "google_search",
    "query": "adidas",
    "geo_location": "California,United States",
    "parse": true
}
TargetSource (Scraping URL)Source (Using Query)

google

google_search,

google_ads,

google_images,

google_lens,

google_maps,

google_travel_hotels,

google_suggest,

google_trends_explore

bing

bing_search

universal

Not available

E-Commerce Scraper API

Getting started

Select the online marketplace you want to scrape: Amazon, Google Shopping, Walmart, Best Buy, Etsy, Target, Other Targets.

Request sample

curl 'https://realtime.oxylabs.io/v1/queries' \
--user "USERNAME:PASSWORD" \
-H "Content-Type: application/json" \
-d '{
        "source": "amazon_product",
        "query": "B07FZ8S74R",
        "geo_location": "90210",
        "parse": true
    }'

We use synchronous Realtime integration method in our examples. If you would like to use Proxy Endpoint or asynchronous Push-Pull integration, refer to the integration methods section.

Forming a request

  1. Pick your integration method: synchronous (Realtime, Proxy Endpoint) or asynchronous (Push-Pull).

  2. When forming a request, include the following elements:

  1. Endpoint:

POST https://realtime.oxylabs.io/v1/queries
  1. Username and password (HTTP authentication). Create API user credentials either during your trial sign-up or product purchase.

USERNAME:PASSWORD

If you need more than one API user for your account, please contact our customer support or message our 24/7 live chat support.

  1. Content-type. When submitting jobs, always add this header:

Content-Type: application/json
  1. Payload:

    1. source - This parameter sets the scraper that will be used to process your request.

    2. URL or query - Provide the URL or query for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.

    3. Additional parameters: Optionally, you can include additional parameters such as geo_location, user_agent_type, parse, render and more to customize your scraping request.

{
    "source": "amazon_product",
    "query": "B07FZ8S74R",
    "geo_location": "90210",
    "parse": true
}
TargetSource (Scraping URL)Source (Using Query)

amazon

amazon_product,

amazon_search,

amazon_pricing,

amazon_sellers,

amazon_bestsellers,

amazon_reviews,

amazon_questions

google_shopping

google_shopping_product,

google_shopping_search,

google_shopping_pricing

universal_ecommerce

Not available

universal_ecommerce

Not available

universal_ecommerce

Not available

universal_ecommerce

Not available

universal_ecommerce

Not available

WEB Scraper API

Getting started

Scrape any URL with our universal source. You can also add additional parameters.

Request sample

curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "universal",
        "url": "https://sandbox.oxylabs.io/"
    }'