Documentation has been updated: see help center and changelog in one place.
Explore
LogoLogo
Oxylabs dashboardProduct
English
  • Documentation
  • Help center
  • Changelog
English
  • Overview
  • PROXIES
    • Integration Guides
      • Get IP Address for Integrations
      • Residential Proxies Guides
        • AdsPower
        • Android
        • ClonBrowser
        • Dolphin Anty
        • Ghost Browser
        • GoLogin
        • Helium Scraper
        • Incogniton
        • iOS
        • Kameleo
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Multilogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • ParseHub
        • Playwright
        • Puppeteer
        • Selenium
        • SEO Neo
        • SessionBox
        • Shadowrocket
        • Super Proxy
        • SwitchyOmega
        • Ubuntu
        • VMLogin
        • WebHarvy
        • Hidemyacc
      • ISP Proxies Guides
        • AdsPower
        • Android
        • Dolphin Anty
        • FoxyProxy
        • GoLogin
        • Incogniton
        • iOS
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Multilogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • SEO Neo
        • Shadowrocket
        • Sphere
        • Super Proxy
        • SwitchyOmega
        • Ubuntu
        • Hidemyacc
      • Mobile Proxies Guides
        • AdsPower
        • Android
        • ClonBrowser
        • Dolphin Anty
        • Ghost Browser
        • GoLogin
        • Helium Scraper
        • Incogniton
        • iOS
        • Kameleo
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Multilogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • ParseHub
        • Playwright
        • Puppeteer
        • Selenium
        • SEO Neo
        • SessionBox
        • Shadowrocket
        • SwitchyOmega
        • Ubuntu
        • VMLogin
        • WebHarvy
      • Dedicated Datacenter Proxies Guides
        • Enterprise
          • Dolphin Anty
          • FoxyProxy
          • GoLogin
          • Lalicat Browser
          • MoreLogin
          • MuLogin
          • Nstbrowser
          • Octoparse
          • Oxy® Proxy Extension for Chrome
          • SEO Neo
          • Shadowrocket
          • Sphere
          • Super Proxy
          • SwitchyOmega
          • Ubuntu
          • Hidemyacc
        • Self-Service
          • Android
          • Dolphin Anty
          • FoxyProxy
          • GoLogin
          • iOS
          • Lalicat Browser
          • MacOS
          • MoreLogin
          • MuLogin
          • Nstbrowser
          • Octoparse
          • Oxy® Proxy Extension for Chrome
          • SEO Neo
          • Shadowrocket
          • Sphere
          • Super Proxy
          • SwitchyOmega
          • Ubuntu
          • Hidemyacc
      • Datacenter Proxies Guides
        • AdsPower
        • Android
        • Dolphin Anty
        • GoLogin
        • iOS
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • SEO Neo
        • Shadowrocket
        • Super Proxy
        • SwitchyOmega
        • Ubuntu
        • Hidemyacc
    • Residential Proxies
      • Getting Started
      • Making Requests
        • Entry Node for China
      • Location Settings
        • Country
        • City
        • State
        • Continent
        • ZIP/Postal code
        • Coordinates
        • ASN Targeting
      • Session Control
        • Sticky Proxy Entry Nodes
      • Protocols
      • Whitelisting IPs
        • Requests with Whitelisted IPs
      • Endpoint Generator
      • Restricted Targets
      • Public API
      • Response Codes
    • ISP Proxies
      • Making Requests
      • Proxy List
      • Proxy Rotation
      • Location Settings
      • Protocols
      • Whitelisting IPs
      • Response Codes
      • Restricted Targets
      • Fair usage policy
    • Mobile Proxies
      • Getting Started
      • Making Requests
        • Entry Node for China
      • Location Settings
        • Country
        • City
        • State
        • Continent
        • Coordinates
        • ASN Targeting
      • Session Control
        • Sticky Proxy Entry Nodes
      • Protocols
      • Whitelisting IPs
      • Endpoint Generator
      • Restricted Targets
      • Public API
      • Response Codes
    • Datacenter Proxies
      • Proxy List
      • IP Control
      • Select Country
      • Protocols
      • Whitelisting
      • Response Codes
      • Restricted Targets
      • Fair usage policy
      • Free Datacenter IPs
    • Dedicated Datacenter Proxies
      • Enterprise
        • Getting Started
        • Proxy List
        • Making Requests
        • Protocols
        • Whitelisting IPs
          • Dashboard
          • RESTful
            • Getting Whitelisted IPs List
            • Adding a Whitelisted IP
            • Removing a Whitelisted IP
            • Saving Changes (5min Cooldown)
        • Datacenter Proxy API
        • Proxy Rotator - Optional
        • Response Codes
      • Self-Service
        • Getting Started
        • Making Requests
        • Proxy List
        • Proxy Rotation
        • Location Settings
        • Protocols
        • Whitelisting IPs
        • Response Codes
        • Restricted Targets
        • Fair usage policy
    • Dedicated ISP Proxies
      • Getting Started
      • Proxy List
      • Making Requests
      • Protocols
      • Whitelisting IPs (RESTful)
        • Getting Whitelisted IPs List
        • Adding a Whitelisted IP
        • Removing a Whitelisted IP
        • Saving Changes (5min Cooldown)
      • Proxy API
      • Proxy Rotator - Optional
      • Response Codes
    • (LEGACY) Shared Datacenter Proxies
      • Getting Started
      • Making Requests
      • Select Country
      • Session Control
      • Response Codes
      • Restricted Targets
  • Advanced proxy solutions
    • Web Unblocker
      • Getting Started
      • Making Requests
        • Session
        • Geo-location
        • Headers & Cookies
        • Custom status code
        • POST requests
      • Headless Browser
        • JavaScript rendering
        • Browser instructions (Beta)
          • List of instructions
      • Sample Response
      • Response Codes
      • Rate Limits
      • Migration Guides
        • From Bright Data Web Unlocker
      • Usage Statistics
      • Billing Information
  • Scraper APIs
    • Web Scraper API
      • Integration Methods
        • Realtime
        • Push-Pull
        • Proxy Endpoint
      • Features
        • Localization
          • Proxy Location
          • SERP Localization
          • E-Commerce Localization
          • Domain, Locale, Results Language
        • JS Rendering & Browser Control
          • JavaScript Rendering
          • Browser Instructions
            • List of instructions
        • Result Processing & Storage
          • Dedicated Parsers
          • Custom Parser
            • Getting started
            • Parsing instruction examples
            • List of functions
              • Function examples
          • Download Images
          • Cloud Storage
        • HTTP Context & Job Management
          • Headers, Cookies, Method
          • User Agent Type
          • Client Notes
        • Scheduler
      • Solutions for AI Workflows
        • Model Context Protocol (MCP)
      • Targets
        • Google
          • Search
            • Web Search
            • AI Overviews
            • Image Search
            • News Search
            • Local Search
            • Reverse Image Search
            • Google Suggest
          • Ads Max
          • Shopping
            • Shopping Product
            • Shopping Search
            • Shopping Pricing
          • Trends: Explore
          • Travel: Hotels
          • Lens
          • URL
        • Amazon
          • Product
          • Search
          • Pricing
          • Sellers
          • Best Sellers
          • Reviews
          • Questions & Answers
          • URL
        • YouTube
          • YouTube Scraping Guide for AI
          • YouTube Search
          • YouTube Video Trainability
          • YouTube Metadata
          • YouTube Downloader
          • YouTube Transcript
        • Generic Target
        • Walmart
          • Search
          • Product
        • Ebay
        • Etsy
          • Search
          • Product
        • Bing
          • Search
          • URL
        • North American E-Commerce
          • Best Buy
            • Search
            • Product
          • Target
            • Search
            • Product
            • Category
          • Kroger
            • Product
            • Search
            • URL
          • Bed Bath & Beyond
          • Menards
          • Petco
          • Staples
          • Grainger
          • Instacart
        • European E-Commerce
          • Allegro
            • Search
            • Product
          • Idealo
          • Mediamarkt
          • Cdiscount
        • Asian E-Commerce
          • Alibaba
          • Aliexpress
          • Lazada
          • Rakuten
          • Tokopedia
          • Flipkart
          • Avnet
          • Indiamart
        • Latin American E-Commerce
          • Mercado Livre
          • Magazine Luiza
          • Falabella
          • Dcard
      • Restricted Targets
      • Response Codes
    • OxyCopilot (Beta)
    • Usage and Billing
      • Usage Statistics
      • Traffic and Billing
      • Rate Limits
  • Dashboard
    • Teams
    • Billing Information
      • Accessing Billing Information
      • Managing Payment Methods
      • Updating Billing Information
      • Canceling a Subscription
    • IP Replacement
  • Guides for Scraper APIs
    • Python SDK
    • Go SDK
    • Forming Requests
    • Forming URLs
    • Using Postman
  • Useful links
    • Oxylabs Dashboard
    • Release Notes
    • Network status
    • Open Source Tools
      • Oxy Parser
      • Oxy Mouse
      • Web Scraper API Scheduler
    • Discord Community
    • GitHub
    • Scraping Experts
  • SUPPORT
    • FAQ
    • Have a Question?
Powered by GitBook
On this page
  • Search Engines
  • Getting started
  • Request sample
  • Forming a request
  • Marketplaces
  • Getting started
  • Request sample
  • Forming a request
  • Other websites
  • Getting started
  • Request sample
  • Forming a request

Was this helpful?

  1. Guides for Scraper APIs

Forming Requests

PreviousIP ReplacementNextForming URLs

Last updated 7 months ago

Was this helpful?

Read detailed guides on how to get started and make requests using for different websites.

Search Engines

Getting started

Select the search engine you want to scrape: , , .

Request sample

curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "google_search",
        "query": "adidas"
    }'
import requests
from pprint import pprint

# Structure payload.
payload = {
    'source': 'google_search',
    'query': 'adidas',
}

# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'),
    json=payload,
)

# Print prettified response to stdout.
pprint(response.json())
const https = require("https");

const username = "USERNAME";
const password = "PASSWORD";
const body = {
    source: "google_search",
    query: "adidas",
};

const options = {
    hostname: "realtime.oxylabs.io",
    path: "/v1/queries",
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        Authorization:
            "Basic " + Buffer.from(`${username}:${password}`).toString("base64"),
    },
};

const request = https.request(options, (response) => {
    let data = "";

    response.on("data", (chunk) => {
        data += chunk;
    });

    response.on("end", () => {
        const responseData = JSON.parse(data);
        console.log(JSON.stringify(responseData, null, 2));
    });
});

request.on("error", (error) => {
    console.error("Error:", error);
});

request.write(JSON.stringify(body));
request.end();
<?php

$params = array(
    'source' => 'google_search',
    'query' => 'adidas',
);

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "USERNAME" . ":" . "PASSWORD");

$headers = array();
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);
using System;
using System.Net.Http;
using System.Net.Http.Json;
using System.Threading.Tasks;

namespace OxyApi
{
    class Program
    {
        static async Task Main()
        {
            const string Username = "USERNAME";
            const string Password = "PASSWORD";

            var parameters = new {
                source = "google_search",
                query = "adidas",
            };

            var client = new HttpClient();

            Uri baseUri = new Uri("https://realtime.oxylabs.io");
            client.BaseAddress = baseUri;

            var requestMessage = new HttpRequestMessage(HttpMethod.Post, "/v1/queries");
            requestMessage.Content = JsonContent.Create(parameters);

            var authenticationString = $"{Username}:{Password}";
            var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes(authenticationString));
            requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString);

            var response = await client.SendAsync(requestMessage);
            var contents = await response.Content.ReadAsStringAsync();

            Console.WriteLine(contents);
        }
    }
}
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
)

func main() {
	const Username = "USERNAME"
	const Password = "PASSWORD"

	payload := map[string]interface{}{
		"source": "google_search",
		"query":  "adidas",
	}

	jsonValue, _ := json.Marshal(payload)

	client := &http.Client{}
	request, _ := http.NewRequest("POST",
		"https://realtime.oxylabs.io/v1/queries",
		bytes.NewBuffer(jsonValue),
	)

	request.SetBasicAuth(Username, Password)
	request.Header.Set("Content-Type", "application/json")
	response, _ := client.Do(request)

	responseText, _ := ioutil.ReadAll(response.Body)
	fmt.Println(string(responseText))
}
https://realtime.oxylabs.io/v1/queries?source=google_search&query=adidas&access_token=12345abcde
package org.example;

import okhttp3.*;
import org.json.JSONObject;
import java.util.concurrent.TimeUnit;

public class Main implements Runnable {
    private static final String AUTHORIZATION_HEADER = "Authorization";
    public static final String USERNAME = "USERNAME";
    public static final String PASSWORD = "PASSWORD";

    public void run() {
        JSONObject jsonObject = new JSONObject();
        jsonObject.put("source", "google_search");
        jsonObject.put("query", "adidas");

        Authenticator authenticator = (route, response) -> {
            String credential = Credentials.basic(USERNAME, PASSWORD);
            return response
                    .request()
                    .newBuilder()
                    .header(AUTHORIZATION_HEADER, credential)
                    .build();
        };

        var client = new OkHttpClient.Builder()
                .authenticator(authenticator)
                .readTimeout(180, TimeUnit.SECONDS)
                .build();

        var mediaType = MediaType.parse("application/json; charset=utf-8");
        var body = RequestBody.create(jsonObject.toString(), mediaType);
        var request = new Request.Builder()
                .url("https://realtime.oxylabs.io/v1/queries")
                .post(body)
                .build();

        try (var response = client.newCall(request).execute()) {
            if (response.body() != null) {
                try (var responseBody = response.body()) {
                    System.out.println(responseBody.string());
                }
            }
        } catch (Exception exception) {
            System.out.println("Error: " + exception.getMessage());
        }

        System.exit(0);
    }

    public static void main(String[] args) {
        new Thread(new Main()).start();
    }
}
{
    "source": "google_search",
    "query": "adidas"
}

We use synchronous integration method in our examples. If you would like to use or asynchronous integration, refer to the section.

Forming a request

  1. Pick your integration method: synchronous (, ) or asynchronous (Push-Pull).

  2. When forming a request, include the following elements:

  1. Endpoint:

POST https://realtime.oxylabs.io/v1/queries
USERNAME:PASSWORD
  1. Content-type. When submitting jobs, always add this header:

Content-Type: application/json
  1. Payload:

    1. source - This parameter sets the scraper that will be used to process your request.

    2. URL or query - Provide the URL or query for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.

{
    "source": "google_search",
    "query": "adidas",
    "geo_location": "California,United States",
    "parse": true
}
  1. Endpoint:

GET realtime.oxylabs.io:60000
  1. Ignore certificates. In cURL, it's -k or --insecure.

USERNAME:PASSWORD
  1. Payload:

    1. URL - Provide the URL for the page you want to scrape.

curl -k -x realtime.oxylabs.io:60000 \
-U "USERNAME:PASSWORD" \
'https://www.google.com/search?q=adidas' \
-H "x-oxylabs-geo-location: California,United States" \
-H "x-oxylabs-parse: 1"
  1. Endpoint:

POST https://data.oxylabs.io/v1/queries
USERNAME:PASSWORD
  1. Content-type. When submitting jobs, always add this header:

Content-Type: application/json
  1. Payload:

    1. source - This parameter sets the scraper that will be used to process your request.

    2. URL or query - Provide the URL or query for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.

{
    "source": "google_search",
    "query": "adidas",
    "geo_location": "California,United States",
    "parse": true
}
  1. Upon submitting a request, you will promptly receive a JSON response containing all job details, including job parameters, job ID, and URLs for downloading job results:

GET https://data.oxylabs.io/v1/queries/{job_id}/results
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=raw
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=parsed
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=png
Target
Source (Scraping URL)
Source (Using Query)

google

google_search,

google_ads,

google_images,

google_lens,

google_maps,

google_travel_hotels,

google_suggest,

google_trends_explore

bing

bing_search

universal

Using query parameter is not supported

Marketplaces

Getting started

Request sample

curl 'https://realtime.oxylabs.io/v1/queries' \
--user "USERNAME:PASSWORD" \
-H "Content-Type: application/json" \
-d '{
        "source": "amazon_product",
        "query": "B07FZ8S74R",
        "geo_location": "90210",
        "parse": true
    }'
import requests
from pprint import pprint

# Structure payload.
payload = {
    'source': 'amazon_product',
    'query': 'B07FZ8S74R',
    'geo_location': '90210',
    'parse': True
}

# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'),
    json=payload,
)

# Print prettified response to stdout.
pprint(response.json())
const https = require("https");

const username = "USERNAME";
const password = "PASSWORD";
const body = {
    source: "amazon_product",
    query: "B07FZ8S74R",
    geo_location: "90210",
    parse: true,
};

const options = {
    hostname: "realtime.oxylabs.io",
    path: "/v1/queries",
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        Authorization:
            "Basic " + Buffer.from(`${username}:${password}`).toString("base64"),
    },
};

const request = https.request(options, (response) => {
    let data = "";

    response.on("data", (chunk) => {
        data += chunk;
    });

    response.on("end", () => {
        const responseData = JSON.parse(data);
        console.log(JSON.stringify(responseData, null, 2));
    });
});

request.on("error", (error) => {
    console.error("Error:", error);
});

request.write(JSON.stringify(body));
request.end();
<?php

$params = array(
    'source' => 'amazon_product',
    'query' => 'B07FZ8S74R',
    'geo_location' => '90210',
    'parse' => true,
);

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "USERNAME" . ":" . "PASSWORD");

$headers = array();
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);
?>
using System;
using System.Net.Http;
using System.Net.Http.Json;
using System.Threading.Tasks;

namespace OxyApi
{
    class Program
    {
        static async Task Main()
        {
            const string Username = "USERNAME";
            const string Password = "PASSWORD";

            var parameters = new {
                source = "amazon_product",
                query = "B07FZ8S74R",
                geo_location = "90210",
                parse = true
            };

            var client = new HttpClient();

            Uri baseUri = new Uri("https://realtime.oxylabs.io");
            client.BaseAddress = baseUri;

            var requestMessage = new HttpRequestMessage(HttpMethod.Post, "/v1/queries");
            requestMessage.Content = JsonContent.Create(parameters);

            var authenticationString = $"{Username}:{Password}";
            var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes(authenticationString));
            requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString);

            var response = await client.SendAsync(requestMessage);
            var contents = await response.Content.ReadAsStringAsync();

            Console.WriteLine(contents);
        }
    }
}
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
)

func main() {
	const Username = "USERNAME"
	const Password = "PASSWORD"

	payload := map[string]interface{}{
		"source":       "amazon_product",
		"query":        "B07FZ8S74R",
		"geo_location": "90210",
		"parse":        true,
	}

	jsonValue, _ := json.Marshal(payload)

	client := &http.Client{}
	request, _ := http.NewRequest("POST",
		"https://realtime.oxylabs.io/v1/queries",
		bytes.NewBuffer(jsonValue),
	)

	request.SetBasicAuth(Username, Password)
	request.Header.Set("Content-Type", "application/json")
	response, _ := client.Do(request)

	responseText, _ := ioutil.ReadAll(response.Body)
	fmt.Println(string(responseText))
}
https://realtime.oxylabs.io/v1/queries?source=amazon_product&query=B07FZ8S74R&geo_location=90210&parse=true&access_token=12345abcde
package org.example;

import okhttp3.*;
import org.json.JSONObject;
import java.util.concurrent.TimeUnit;

public class Main implements Runnable {
    private static final String AUTHORIZATION_HEADER = "Authorization";
    public static final String USERNAME = "USERNAME";
    public static final String PASSWORD = "PASSWORD";

    public void run() {
        JSONObject jsonObject = new JSONObject();
        jsonObject.put("source", "amazon_product");
        jsonObject.put("query", "B07FZ8S74R");
        jsonObject.put("geo_location", "90210");
        jsonObject.put("parse", true);

        Authenticator authenticator = (route, response) -> {
            String credential = Credentials.basic(USERNAME, PASSWORD);
            return response
                    .request()
                    .newBuilder()
                    .header(AUTHORIZATION_HEADER, credential)
                    .build();
        };

        var client = new OkHttpClient.Builder()
                .authenticator(authenticator)
                .readTimeout(180, TimeUnit.SECONDS)
                .build();

        var mediaType = MediaType.parse("application/json; charset=utf-8");
        var body = RequestBody.create(jsonObject.toString(), mediaType);
        var request = new Request.Builder()
                .url("https://realtime.oxylabs.io/v1/queries")
                .post(body)
                .build();

        try (var response = client.newCall(request).execute()) {
            if (response.body() != null) {
                try (var responseBody = response.body()) {
                    System.out.println(responseBody.string());
                }
            }
        } catch (Exception exception) {
            System.out.println("Error: " + exception.getMessage());
        }

        System.exit(0);
    }

    public static void main(String[] args) {
        new Thread(new Main()).start();
    }
}
{
    "source": "amazon_product",
    "query": "B07FZ8S74R",
    "geo_location": "90210",
    "parse": true
}

Forming a request

  1. When forming a request, include the following elements:

  1. Endpoint:

POST https://realtime.oxylabs.io/v1/queries
USERNAME:PASSWORD
  1. Content-type. When submitting jobs, always add this header:

Content-Type: application/json
  1. Payload:

    1. source - This parameter sets the scraper that will be used to process your request.

    2. URL or query - Provide the URL or query for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.

{
    "source": "amazon_product",
    "query": "B07FZ8S74R",
    "geo_location": "90210",
    "parse": true
}
  1. Endpoint:

GET realtime.oxylabs.io:60000
  1. Ignore certificates. In cURL, it's -k or --insecure.

USERNAME:PASSWORD
  1. Payload:

    1. URL - Provide the URL for the page you want to scrape.

curl -k -x realtime.oxylabs.io:60000 \
-U "USERNAME:PASSWORD" \
'https://www.amazon.com/dp/b07fz8s74r' \
-H "x-oxylabs-geo-location: 90210" \
-H "x-oxylabs-parse: 1"
  1. Endpoint:

POST https://data.oxylabs.io/v1/queries
USERNAME:PASSWORD
  1. Content-type. When submitting jobs, always add this header:

Content-Type: application/json
  1. Payload:

    1. source - This parameter sets the scraper that will be used to process your request.

    2. URL or query - Provide the URL or query for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.

{
    "source": "amazon_product",
    "query": "B07FZ8S74R",
    "geo_location": "90210",
    "parse": true
}
  1. Upon submitting a request, you will promptly receive a JSON response containing all job details, including job parameters, job ID, and URLs for downloading job results:

GET https://data.oxylabs.io/v1/queries/{job_id}/results
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=raw
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=parsed
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=pn
Target
Source (Scraping URL)
Source (Using Query)

amazon

amazon_product,

amazon_search,

amazon_pricing,

amazon_sellers,

amazon_bestsellers,

amazon_reviews,

amazon_questions

google

google_shopping_product,

google_shopping_search,

google_shopping_pricing

universal

Using query parameter is not supported

Other websites

Getting started

Request sample

curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "universal",
        "url": "https://sandbox.oxylabs.io/"
    }'
import requests
from pprint import pprint


# Structure payload.
payload = {
    'source': 'universal',
    'url': 'https://sandbox.oxylabs.io/',
    # 'render': 'html', # If page type requires
}

# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'), # Your credentials go here
    json=payload,
)

# Instead of response with job status and results url, this will return the
# JSON response with results.
pprint(response.json())
const https = require("https");

const username = "USERNAME";
const password = "PASSWORD";
const body = {
    source: "universal",
    url: "https://sandbox.oxylabs.io/",
    // 'render': 'html' // If page type requires
};

const options = {
    hostname: "realtime.oxylabs.io",
    path: "/v1/queries",
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        Authorization: "Basic " + Buffer.from(`${username}:${password}`).toString("base64"),
    },
};

const request = https.request(options, (response) => {
    let data = "";

    response.on("data", (chunk) => {
        data += chunk;
    });

    response.on("end", () => {
        const responseData = JSON.parse(data);
        console.log(JSON.stringify(responseData, null, 2));
    });
});

request.on("error", (error) => {
    console.error("Error:", error);
});

request.write(JSON.stringify(body));
request.end();
# The whole string you submit has to be URL-encoded.

https://realtime.oxylabs.io/v1/queries?source=universal&url=https%3A%2F%2Fsandbox.oxylabs.io%2F&access_token=12345abcde
<?php

$params = array(
    'source' => 'universal',
    'url' => 'https://sandbox.oxylabs.io/',
    // 'render' => 'html' // If page type requires
);

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "USERNAME" . ":" . "PASSWORD");

$headers = array();
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
)

func main() {
	const Username = "USERNAME"
	const Password = "PASSWORD"

	payload := map[string]interface{}{
		"source": "universal",
		"url":    "https://sandbox.oxylabs.io/",
		// "render": "html" // If page type requires
	}

	jsonValue, _ := json.Marshal(payload)

	client := &http.Client{}
	request, _ := http.NewRequest("POST",
		"https://realtime.oxylabs.io/v1/queries",
		bytes.NewBuffer(jsonValue),
	)

	request.SetBasicAuth(Username, Password)
	response, _ := client.Do(request)

	responseText, _ := ioutil.ReadAll(response.Body)
	fmt.Println(string(responseText))
}
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Net.Http.Json;
using System.Threading.Tasks;

namespace OxyApi
{
    class Program
    {
        static async Task Main()
        {
            const string Username = "USERNAME";
            const string Password = "PASSWORD";

            var parameters = new {
                source = "universal",
                url = "https://sandbox.oxylabs.io/",
                // render = "html", // If page type requires
            };

            var client = new HttpClient();

            Uri baseUri = new Uri("https://realtime.oxylabs.io");
            client.BaseAddress = baseUri;

            var requestMessage = new HttpRequestMessage(HttpMethod.Post, "/v1/queries");
            requestMessage.Content = JsonContent.Create(parameters);

            var authenticationString = $"{Username}:{Password}";
            var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.ASCIIEncoding.UTF8.GetBytes(authenticationString));
            requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString);

            var response = await client.SendAsync(requestMessage);
            var contents = await response.Content.ReadAsStringAsync();

            Console.WriteLine(contents);
        }
    }
}
package org.example;

import okhttp3.*;
import org.json.JSONObject;
import java.util.concurrent.TimeUnit;

public class Main implements Runnable {
    private static final String AUTHORIZATION_HEADER = "Authorization";
    public static final String USERNAME = "USERNAME";
    public static final String PASSWORD = "PASSWORD";

    public void run() {
        JSONObject jsonObject = new JSONObject();
        jsonObject.put("source", "universal");
        jsonObject.put("url", "https://sandbox.oxylabs.io/");

        Authenticator authenticator = (route, response) -> {
            String credential = Credentials.basic(USERNAME, PASSWORD);
            return response
                    .request()
                    .newBuilder()
                    .header(AUTHORIZATION_HEADER, credential)
                    .build();
        };

        var client = new OkHttpClient.Builder()
                .authenticator(authenticator)
                .readTimeout(180, TimeUnit.SECONDS)
                .build();

        var mediaType = MediaType.parse("application/json; charset=utf-8");
        var body = RequestBody.create(jsonObject.toString(), mediaType);
        var request = new Request.Builder()
                .url("https://realtime.oxylabs.io/v1/queries")
                .post(body)
                .build();

        try (var response = client.newCall(request).execute()) {
            if (response.body() != null) {
                try (var responseBody = response.body()) {
                    System.out.println(responseBody.string());
                }
            }
        } catch (Exception exception) {
            System.out.println("Error: " + exception.getMessage());
        }

        System.exit(0);
    }

    public static void main(String[] args) {
        new Thread(new Main()).start();
    }
}
{
    "source": "universal", 
    "url": "https://sandbox.oxylabs.io/"
}

Forming a request

  1. When forming a request, include the following elements:

  1. Endpoint:

POST https://realtime.oxylabs.io/v1/queries
USERNAME:PASSWORD
  1. Content-type. When submitting jobs, always add this header:

Content-Type: application/json
  1. Payload.

    1. source - This parameter sets the scraper that will be used to process your request.

    2. URL - Provide the URL of the target you want to scrape, for example:

      1. Real Estate: Idealista, Redfin, Zillow, Zoopla

      2. Travel: Airbnb, Agoda, Booking, TripAdvisor

      3. Automotive: Crunchbase, ZoomInfo, AngelList, Product Hunt

      4. Company data: Netflix, SoundCloud, YouTube, IMDb

      5. Entertainment: AutoEurope, Autotrader, RockAuto, Halfords

      6. Any other.

    {
        "source": "universal",
        "url": "https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/"
    }
  1. Endpoint:

GET realtime.oxylabs.io:60000
  1. Ignore certificates. In cURL, it's -k or --insecure.

USERNAME:PASSWORD
  1. Payload:

    1. URL - Provide the URL for the page you want to scrape, for example:

      1. Real Estate: Idealista, Redfin, Zillow, Zoopla

      2. Travel: Airbnb, Agoda, Booking, TripAdvisor

      3. Automotive: Crunchbase, ZoomInfo, AngelList, Product Hunt

      4. Company data: Netflix, SoundCloud, YouTube, IMDb

      5. Entertainment: AutoEurope, Autotrader, RockAuto, Halfords

      6. Any other.

curl -k -x realtime.oxylabs.io:60000 \
-U "USERNAME:PASSWORD" \
'https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/' 
  1. Endpoint:

POST https://data.oxylabs.io/v1/queries
USERNAME:PASSWORD
  1. Content-type. When submitting jobs, always add this header:

Content-Type: application/json
  1. Payload.

    1. source - This parameter sets the scraper that will be used to process your request.

    2. URL - Provide the URL of the target you want to scrape, for example:

      1. Real Estate: Idealista, Redfin, Zillow, Zoopla

      2. Travel: Airbnb, Agoda, Booking, TripAdvisor

      3. Automotive: Crunchbase, ZoomInfo, AngelList, Product Hunt

      4. Company data: Netflix, SoundCloud, YouTube, IMDb

      5. Entertainment: AutoEurope, Autotrader, RockAuto, Halfords

      6. Any other.

    {
        "source": "universal",
        "url": "https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/"
    }
  2. Upon submitting a request, you will promptly receive a JSON response containing all job details, including job parameters, job ID, and URLs for downloading job results:

GET https://data.oxylabs.io/v1/queries/{job_id}/results
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=raw
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=parsed
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=pn

If you need any assistance in making your first request, feel free to contact us via the 24/7 available live chat.

All information herein is provided on an “as is” basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on this page. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website’s terms of service or receive a scraping license.

Username and password (). Create API user credentials either during your .

If you need more than one API user for your account, please contact our or message our 24/7 live chat support.

: Optionally, you can include additional parameters such as geo_location, user_agent_type, parse, render and more to customize your scraping request.

Username and password (). Create API user credentials either during your .

If you need more than one API user for your account, please contact our or message our 24/7 live chat support.

: Optionally, you can include additional parameters such as geo_location, user_agent_type, parse, and send them as headers.

Username and password (). Create API user credentials either during your .

If you need more than one API user for your account, please contact our or message our 24/7 live chat support.

: Optionally, you can include additional parameters such as geo_location, user_agent_type, parse, and more to customize your scraping request.

Select the online marketplace you want to scrape: , , , Best Buy, Etsy, , .

We use synchronous integration method in our examples. If you would like to use or asynchronous integration, refer to the section.

Pick your integration method: synchronous (, ) or asynchronous (Push-Pull).

Username and password (). Create API user credentials either during your .

If you need more than one API user for your account, please contact our or message our 24/7 live chat support.

: Optionally, you can include additional parameters such as geo_location, user_agent_type, parse, render and more to customize your scraping request.

Username and password (). Create API user credentials either during your .

If you need more than one API user for your account, please contact our or message our 24/7 live chat support.

: Optionally, you can include additional parameters such as geo_location, user_agent_type, parse, and send them as headers.

Username and password (). Create API user credentials either during your .

If you need more than one API user for your account, please contact our or message our 24/7 live chat support.

: Optionally, you can include additional parameters such as geo_location, user_agent_type, parse, and more to customize your scraping request.

Scrape any URL with our universal source. You can also add .

We use synchronous integration method in our examples. If you would like to use or asynchronous integration, refer to the section.

Pick your integration method: synchronous (, ) or asynchronous (Push-Pull).

Username and password (). Create API user credentials either during your .

If you need more than one API user for your account, please contact our or message our 24/7 live chat support.

: Optionally, you can include additional parameters such as geo_location, user_agent_type, and more to customize your scraping request.

Username and password (). Create API user credentials either during your .

If you need more than one API user for your account, please contact our or message our 24/7 live chat support.

: Optionally, you can include additional parameters such as geo_location, user_agent_type, and send them as headers.

Username and password (). Create API user credentials either during your .

If you need more than one API user for your account, please contact our or message our 24/7 live chat support.

: Optionally, you can include additional parameters such as geo_location, user_agent_type, and more to customize your scraping request.

Web Scraper API
Google
Bing
Other Search Engines
Realtime
Proxy Endpoint
Push-Pull
integration methods
Realtime
Proxy Endpoint
HTTP authentication
trial sign-up or product purchase
customer support
Additional parameters
HTTP authentication
trial sign-up or product purchase
customer support
Additional parameters
HTTP authentication
trial sign-up or product purchase
customer support
Additional parameters
Amazon
Google Shopping
Walmart
Target
Other Websites
Realtime
Proxy Endpoint
Push-Pull
integration methods
Realtime
Proxy Endpoint
HTTP authentication
trial sign-up or product purchase
customer support
Additional parameters
HTTP authentication
trial sign-up or product purchase
customer support
Additional parameters
HTTP authentication
trial sign-up or product purchase
customer support
Additional parameters
additional parameters
Realtime
Proxy Endpoint
Push-Pull
integration methods
Realtime
Proxy Endpoint
HTTP authentication
trial sign-up or product purchase
customer support
Additional parameters
HTTP authentication
trial sign-up or product purchase
customer support
Additional parameters
HTTP authentication
trial sign-up or product purchase
customer support
Additional parameters
Google
Bing
Other Search Engines
Amazon
Google Shopping
Best Buy
Etsy
Walmart
Target
Other Websites