Forming Requests

Read detailed guides on how to get started and make requests using Web Scraper API for different websites.

Search Engines

Getting started

Select the search engine you want to scrape: Google, Bing, Other Search Engines.

Request sample

curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "google_search",
        "query": "adidas"
    }'

We use synchronous Realtime integration method in our examples. If you would like to use Proxy Endpoint or asynchronous Push-Pull integration, refer to the integration methods section.

Forming a request

  1. Pick your integration method: synchronous (Realtime, Proxy Endpoint) or asynchronous (Push-Pull).

  2. When forming a request, include the following elements:

  1. Endpoint:

POST https://realtime.oxylabs.io/v1/queries
  1. Username and password (HTTP authentication). Create API user credentials either during your trial sign-up or product purchase.

USERNAME:PASSWORD

If you need more than one API user for your account, please contact our customer support or message our 24/7 live chat support.

  1. Content-type. When submitting jobs, always add this header:

Content-Type: application/json
  1. Payload:

    1. source - This parameter sets the scraper that will be used to process your request.

    2. URL or query - Provide the URL or query for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.

    3. Additional parameters: Optionally, you can include additional parameters such as geo_location, user_agent_type, parse, render and more to customize your scraping request.

{
    "source": "google_search",
    "query": "adidas",
    "geo_location": "California,United States",
    "parse": true
}

Marketplaces

Getting started

Select the online marketplace you want to scrape: Amazon, Google Shopping, Walmart, Best Buy, Etsy, Target, Other Websites.

Request sample

curl 'https://realtime.oxylabs.io/v1/queries' \
--user "USERNAME:PASSWORD" \
-H "Content-Type: application/json" \
-d '{
        "source": "amazon_product",
        "query": "B07FZ8S74R",
        "geo_location": "90210",
        "parse": true
    }'

We use synchronous Realtime integration method in our examples. If you would like to use Proxy Endpoint or asynchronous Push-Pull integration, refer to the integration methods section.

Forming a request

  1. Pick your integration method: synchronous (Realtime, Proxy Endpoint) or asynchronous (Push-Pull).

  2. When forming a request, include the following elements:

  1. Endpoint:

POST https://realtime.oxylabs.io/v1/queries
  1. Username and password (HTTP authentication). Create API user credentials either during your trial sign-up or product purchase.

USERNAME:PASSWORD

If you need more than one API user for your account, please contact our customer support or message our 24/7 live chat support.

  1. Content-type. When submitting jobs, always add this header:

Content-Type: application/json
  1. Payload:

    1. source - This parameter sets the scraper that will be used to process your request.

    2. URL or query - Provide the URL or query for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.

    3. Additional parameters: Optionally, you can include additional parameters such as geo_location, user_agent_type, parse, render and more to customize your scraping request.

{
    "source": "amazon_product",
    "query": "B07FZ8S74R",
    "geo_location": "90210",
    "parse": true
}

Other websites

Getting started

Scrape any URL with our universal source. You can also add additional parameters.

Request sample

curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "universal",
        "url": "https://sandbox.oxylabs.io/"
    }'