Forming Requests

Read detailed guides on how to get started and make requests using Web Scraper API for different websites.

Search Engines

Getting started

Select the search engine you want to scrape: Google, Bing, Other Search Engines.

Request sample

curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "google_search",
        "query": "adidas"
    }'

We use synchronous Realtime integration method in our examples. If you would like to use Proxy Endpoint or asynchronous Push-Pull integration, refer to the integration methods section.

Forming a request

  1. Pick your integration method: synchronous (Realtime, Proxy Endpoint) or asynchronous (Push-Pull).

  2. When forming a request, include the following elements:

  1. Endpoint:

POST https://realtime.oxylabs.io/v1/queries
  1. Username and password (HTTP authentication). Create API user credentials either during your trial sign-up or product purchase.

USERNAME:PASSWORD

If you need more than one API user for your account, please contact our customer support or message our 24/7 live chat support.

  1. Content-type. When submitting jobs, always add this header:

Content-Type: application/json
  1. Payload:

    1. source - This parameter sets the scraper that will be used to process your request.

    2. URL or query - Provide the URL or query for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.

    3. Additional parameters: Optionally, you can include additional parameters such as geo_location, user_agent_type, parse, render and more to customize your scraping request.

{
    "source": "google_search",
    "query": "adidas",
    "geo_location": "California,United States",
    "parse": true
}
Target
Source (Scraping URL)
Source (Using Query)

google

google_search,

google_ads,

google_images,

google_lens,

google_maps,

google_travel_hotels,

google_suggest,

google_trends_explore

bing

bing_search

universal

Using query parameter is not supported

Marketplaces

Getting started

Select the online marketplace you want to scrape: Amazon, Google Shopping, Walmart, Best Buy, Etsy, Target, Other Websites.

Request sample

curl 'https://realtime.oxylabs.io/v1/queries' \
--user "USERNAME:PASSWORD" \
-H "Content-Type: application/json" \
-d '{
        "source": "amazon_product",
        "query": "B07FZ8S74R",
        "geo_location": "90210",
        "parse": true
    }'

We use synchronous Realtime integration method in our examples. If you would like to use Proxy Endpoint or asynchronous Push-Pull integration, refer to the integration methods section.

Forming a request

  1. Pick your integration method: synchronous (Realtime, Proxy Endpoint) or asynchronous (Push-Pull).

  2. When forming a request, include the following elements:

  1. Endpoint:

POST https://realtime.oxylabs.io/v1/queries
  1. Username and password (HTTP authentication). Create API user credentials either during your trial sign-up or product purchase.

USERNAME:PASSWORD

If you need more than one API user for your account, please contact our customer support or message our 24/7 live chat support.

  1. Content-type. When submitting jobs, always add this header:

Content-Type: application/json
  1. Payload:

    1. source - This parameter sets the scraper that will be used to process your request.

    2. URL or query - Provide the URL or query for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.

    3. Additional parameters: Optionally, you can include additional parameters such as geo_location, user_agent_type, parse, render and more to customize your scraping request.

{
    "source": "amazon_product",
    "query": "B07FZ8S74R",
    "geo_location": "90210",
    "parse": true
}
Target
Source (Scraping URL)
Source (Using Query)

amazon

amazon_product,

amazon_search,

amazon_pricing,

amazon_sellers,

amazon_bestsellers,

amazon_reviews,

amazon_questions

google

google_shopping_product,

google_shopping_search,

google_shopping_pricing

universal

Using query parameter is not supported

Other websites

Getting started

Scrape any URL with our universal source. You can also add additional parameters.

Request sample

curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "universal",
        "url": "https://sandbox.oxylabs.io/"
    }'

We use synchronous Realtime integration method in our examples. If you would like to use Proxy Endpoint or asynchronous Push-Pull integration, refer to the integration methods section.

Forming a request

  1. Pick your integration method: synchronous (Realtime, Proxy Endpoint) or asynchronous (Push-Pull).

  2. When forming a request, include the following elements:

  1. Endpoint:

POST https://realtime.oxylabs.io/v1/queries
  1. Username and password (HTTP authentication). Create API user credentials either during your trial sign-up or product purchase.

USERNAME:PASSWORD

If you need more than one API user for your account, please contact our customer support or message our 24/7 live chat support.

  1. Content-type. When submitting jobs, always add this header:

Content-Type: application/json
  1. Payload.

    1. source - This parameter sets the scraper that will be used to process your request.

    2. URL - Provide the URL of the target you want to scrape, for example:

      1. Real Estate: Idealista, Redfin, Zillow, Zoopla

      2. Travel: Airbnb, Agoda, Booking, TripAdvisor

      3. Automotive: Crunchbase, ZoomInfo, AngelList, Product Hunt

      4. Company data: Netflix, SoundCloud, YouTube, IMDb

      5. Entertainment: AutoEurope, Autotrader, RockAuto, Halfords

      6. Any other.

    3. Additional parameters: Optionally, you can include additional parameters such as geo_location, user_agent_type, and more to customize your scraping request.

    {
        "source": "universal",
        "url": "https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/"
    }

If you need any assistance in making your first request, feel free to contact us via the 24/7 available live chat.

All information herein is provided on an “as is” basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on this page. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website’s terms of service or receive a scraping license.

Last updated