# Forming Requests

Read detailed guides on how to get started and make requests using [**Web Scraper API**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api) for different websites.

## Search Engines

### Getting started

Select the search engine you want to scrape: [**Google**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/targets/google), [**Bing**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/targets/bing), [**Other Search Engines**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/targets/generic-target).

### Request sample

{% tabs %}
{% tab title="cURL" %}

```shell
curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "google_search",
        "query": "adidas"
    }'
```

{% endtab %}

{% tab title="Python" %}

```python
import requests
from pprint import pprint

# Structure payload.
payload = {
    'source': 'google_search',
    'query': 'adidas',
}

# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'),
    json=payload,
)

# Print prettified response to stdout.
pprint(response.json())
```

{% endtab %}

{% tab title="Node.js" %}

```javascript
const https = require("https");

const username = "USERNAME";
const password = "PASSWORD";
const body = {
    source: "google_search",
    query: "adidas",
};

const options = {
    hostname: "realtime.oxylabs.io",
    path: "/v1/queries",
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        Authorization:
            "Basic " + Buffer.from(`${username}:${password}`).toString("base64"),
    },
};

const request = https.request(options, (response) => {
    let data = "";

    response.on("data", (chunk) => {
        data += chunk;
    });

    response.on("end", () => {
        const responseData = JSON.parse(data);
        console.log(JSON.stringify(responseData, null, 2));
    });
});

request.on("error", (error) => {
    console.error("Error:", error);
});

request.write(JSON.stringify(body));
request.end();
```

{% endtab %}

{% tab title="PHP" %}

```php
<?php

$params = array(
    'source' => 'google_search',
    'query' => 'adidas',
);

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "USERNAME" . ":" . "PASSWORD");

$headers = array();
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);
```

{% endtab %}

{% tab title="C#" %}

```csharp
using System;
using System.Net.Http;
using System.Net.Http.Json;
using System.Threading.Tasks;

namespace OxyApi
{
    class Program
    {
        static async Task Main()
        {
            const string Username = "USERNAME";
            const string Password = "PASSWORD";

            var parameters = new {
                source = "google_search",
                query = "adidas",
            };

            var client = new HttpClient();

            Uri baseUri = new Uri("https://realtime.oxylabs.io");
            client.BaseAddress = baseUri;

            var requestMessage = new HttpRequestMessage(HttpMethod.Post, "/v1/queries");
            requestMessage.Content = JsonContent.Create(parameters);

            var authenticationString = $"{Username}:{Password}";
            var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes(authenticationString));
            requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString);

            var response = await client.SendAsync(requestMessage);
            var contents = await response.Content.ReadAsStringAsync();

            Console.WriteLine(contents);
        }
    }
}
```

{% endtab %}

{% tab title="Golang" %}

```go
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
)

func main() {
	const Username = "USERNAME"
	const Password = "PASSWORD"

	payload := map[string]interface{}{
		"source": "google_search",
		"query":  "adidas",
	}

	jsonValue, _ := json.Marshal(payload)

	client := &http.Client{}
	request, _ := http.NewRequest("POST",
		"https://realtime.oxylabs.io/v1/queries",
		bytes.NewBuffer(jsonValue),
	)

	request.SetBasicAuth(Username, Password)
	request.Header.Set("Content-Type", "application/json")
	response, _ := client.Do(request)

	responseText, _ := ioutil.ReadAll(response.Body)
	fmt.Println(string(responseText))
}
```

{% endtab %}

{% tab title="HTTP" %}

```http
https://realtime.oxylabs.io/v1/queries?source=google_search&query=adidas&access_token=12345abcde
```

{% endtab %}

{% tab title="Java" %}

```java
package org.example;

import okhttp3.*;
import org.json.JSONObject;
import java.util.concurrent.TimeUnit;

public class Main implements Runnable {
    private static final String AUTHORIZATION_HEADER = "Authorization";
    public static final String USERNAME = "USERNAME";
    public static final String PASSWORD = "PASSWORD";

    public void run() {
        JSONObject jsonObject = new JSONObject();
        jsonObject.put("source", "google_search");
        jsonObject.put("query", "adidas");

        Authenticator authenticator = (route, response) -> {
            String credential = Credentials.basic(USERNAME, PASSWORD);
            return response
                    .request()
                    .newBuilder()
                    .header(AUTHORIZATION_HEADER, credential)
                    .build();
        };

        var client = new OkHttpClient.Builder()
                .authenticator(authenticator)
                .readTimeout(180, TimeUnit.SECONDS)
                .build();

        var mediaType = MediaType.parse("application/json; charset=utf-8");
        var body = RequestBody.create(jsonObject.toString(), mediaType);
        var request = new Request.Builder()
                .url("https://realtime.oxylabs.io/v1/queries")
                .post(body)
                .build();

        try (var response = client.newCall(request).execute()) {
            if (response.body() != null) {
                try (var responseBody = response.body()) {
                    System.out.println(responseBody.string());
                }
            }
        } catch (Exception exception) {
            System.out.println("Error: " + exception.getMessage());
        }

        System.exit(0);
    }

    public static void main(String[] args) {
        new Thread(new Main()).start();
    }
}
```

{% endtab %}

{% tab title="JSON" %}

```json
{
    "source": "google_search",
    "query": "adidas"
}
```

{% endtab %}
{% endtabs %}

We use synchronous [**Realtime**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/realtime) integration method in our examples. If you would like to use [**Proxy Endpoint**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/proxy-endpoint) or asynchronous [**Push-Pull**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/push-pull) integration, refer to the [**integration methods**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods) section.

### Forming a request

1. Pick your integration method: synchronous ([**Realtime**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/realtime), [**Proxy Endpoint**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/proxy-endpoint)) or asynchronous  ([**Push-Pull**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/push-pull)).
2. When forming a request, include the following elements:

{% tabs %}
{% tab title="Realtime" %}

1. **Endpoint:**

```
POST https://realtime.oxylabs.io/v1/queries
```

2. **Username and password** ([**HTTP authentication**](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication))**.** Create API user credentials either during your [**trial sign-up or product purchase**](https://dashboard.oxylabs.io/en/registration?productToBuy=SCRAPI_SERP)**.**

```
USERNAME:PASSWORD
```

{% hint style="warning" %}
If you need more than one API user for your account, please contact our [**customer support**](mailto:support@oxylabs.io) or message our 24/7 live chat support.
{% endhint %}

3. **Content-type**. When submitting jobs, always add this header:

```
Content-Type: application/json
```

4. **Payload:**
   1. `source` - This parameter sets the scraper that will be used to process your request.
   2. `URL` or `query` - Provide the `URL` or `query` for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.
   3. [**Additional parameters**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/features): Optionally, you can include additional parameters such as `geo_location`, `user_agent_type`, `parse`, `render` and more to customize your scraping request.

```json
{
    "source": "google_search",
    "query": "adidas",
    "geo_location": "California,United States",
    "parse": true
}
```

{% endtab %}

{% tab title="Proxy Endpoint" %}

1. **Endpoint:**

```
GET realtime.oxylabs.io:60000
```

2. **Ignore certificates.** In `cURL`, it's `-k` or `--insecure`.
3. **Username and password** ([**HTTP authentication**](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication))**.** Create API user credentials either during your [**trial sign-up or product purchase**](https://dashboard.oxylabs.io/en/registration?productToBuy=SCRAPI_SERP)**.**

```
USERNAME:PASSWORD
```

{% hint style="warning" %}
If you need more than one API user for your account, please contact our [**customer support**](mailto:support@oxylabs.io) or message our 24/7 live chat support.
{% endhint %}

4. **Payload:**
   1. `URL` - Provide the `URL` for the page you want to scrape.
   2. [**Additional parameters**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/features): Optionally, you can include additional parameters such as `geo_location`, `user_agent_type`, `parse`, and send them as headers.

```bash
curl -k -x realtime.oxylabs.io:60000 \
-U "USERNAME:PASSWORD" \
'https://www.google.com/search?q=adidas' \
-H "x-oxylabs-geo-location: California,United States" \
-H "x-oxylabs-parse: 1"
```

{% endtab %}

{% tab title="Push-Pull" %}

1. **Endpoint:**

```
POST https://data.oxylabs.io/v1/queries
```

2. **Username and password** ([**HTTP authentication**](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication))**.** Create API user credentials either during your [**trial sign-up or product purchase**](https://dashboard.oxylabs.io/en/registration?productToBuy=SCRAPI_SERP)**.**

```
USERNAME:PASSWORD
```

{% hint style="warning" %}
If you need more than one API user for your account, please contact our [**customer support**](mailto:support@oxylabs.io) or message our 24/7 live chat support.
{% endhint %}

3. **Content-type**. When submitting jobs, always add this header:

```
Content-Type: application/json
```

4. **Payload:**
   1. `source` - This parameter sets the scraper that will be used to process your request.
   2. `URL` or `query` - Provide the `URL` or `query` for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.
   3. [**Additional parameters**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/features): Optionally, you can include additional parameters such as `geo_location`, `user_agent_type`, `parse`, and more to customize your scraping request.

```json
{
    "source": "google_search",
    "query": "adidas",
    "geo_location": "California,United States",
    "parse": true
}
```

5. Upon submitting a request, you will promptly receive a JSON response containing all job details, including job parameters, job ID, and URLs for downloading job results:

```
GET https://data.oxylabs.io/v1/queries/{job_id}/results
```

```
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=raw
```

```
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=parsed
```

```
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=png
```

{% endtab %}
{% endtabs %}

<table><thead><tr><th width="217">Target</th><th width="246">Source (Scraping URL)</th><th>Source (Using Query)</th></tr></thead><tbody><tr><td><a href="../scraping-solutions/web-scraper-api/targets/google"><strong>Google</strong></a></td><td><code>google</code></td><td><p><code>google_search</code>,</p><p><code>google_ads</code>,</p><p><code>google_lens</code>,</p><p><code>google_maps</code>,</p><p><code>google_travel_hotels</code>,</p><p><code>google_suggest</code>,</p><p><code>google_trends_explore</code></p></td></tr><tr><td><a href="../scraping-solutions/web-scraper-api/targets/bing"><strong>Bing</strong></a></td><td><code>bing</code></td><td><code>bing_search</code></td></tr><tr><td><a href="../scraping-solutions/web-scraper-api/targets/generic-target"><strong>Other Search Engines</strong></a></td><td><code>universal</code></td><td>Using <code>query</code> parameter is not supported</td></tr></tbody></table>

## Marketplaces

### Getting started

Select the online marketplace you want to scrape: [**Amazon**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/targets/amazon)**,** [**Google Shopping**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/targets/google)**,** [**Walmart**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/targets/walmart)**,** [**Best Buy**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/targets/north-american-e-commerce/bestbuy)**,** [**Etsy**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/targets/etsy)**,** [**Target**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/targets/north-american-e-commerce/target)**,** [**Other Websites**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/targets/generic-target)**.**

### Request sample

{% tabs %}
{% tab title="cURL" %}

```bash
curl 'https://realtime.oxylabs.io/v1/queries' \
--user "USERNAME:PASSWORD" \
-H "Content-Type: application/json" \
-d '{
        "source": "amazon_product",
        "query": "B07FZ8S74R",
        "geo_location": "90210",
        "parse": true
    }'
```

{% endtab %}

{% tab title="Python" %}

```python
import requests
from pprint import pprint

# Structure payload.
payload = {
    'source': 'amazon_product',
    'query': 'B07FZ8S74R',
    'geo_location': '90210',
    'parse': True
}

# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'),
    json=payload,
)

# Print prettified response to stdout.
pprint(response.json())
```

{% endtab %}

{% tab title="Node.js" %}

```javascript
const https = require("https");

const username = "USERNAME";
const password = "PASSWORD";
const body = {
    source: "amazon_product",
    query: "B07FZ8S74R",
    geo_location: "90210",
    parse: true,
};

const options = {
    hostname: "realtime.oxylabs.io",
    path: "/v1/queries",
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        Authorization:
            "Basic " + Buffer.from(`${username}:${password}`).toString("base64"),
    },
};

const request = https.request(options, (response) => {
    let data = "";

    response.on("data", (chunk) => {
        data += chunk;
    });

    response.on("end", () => {
        const responseData = JSON.parse(data);
        console.log(JSON.stringify(responseData, null, 2));
    });
});

request.on("error", (error) => {
    console.error("Error:", error);
});

request.write(JSON.stringify(body));
request.end();
```

{% endtab %}

{% tab title="PHP" %}

```php
<?php

$params = array(
    'source' => 'amazon_product',
    'query' => 'B07FZ8S74R',
    'geo_location' => '90210',
    'parse' => true,
);

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "USERNAME" . ":" . "PASSWORD");

$headers = array();
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);
?>
```

{% endtab %}

{% tab title="C#" %}

```csharp
using System;
using System.Net.Http;
using System.Net.Http.Json;
using System.Threading.Tasks;

namespace OxyApi
{
    class Program
    {
        static async Task Main()
        {
            const string Username = "USERNAME";
            const string Password = "PASSWORD";

            var parameters = new {
                source = "amazon_product",
                query = "B07FZ8S74R",
                geo_location = "90210",
                parse = true
            };

            var client = new HttpClient();

            Uri baseUri = new Uri("https://realtime.oxylabs.io");
            client.BaseAddress = baseUri;

            var requestMessage = new HttpRequestMessage(HttpMethod.Post, "/v1/queries");
            requestMessage.Content = JsonContent.Create(parameters);

            var authenticationString = $"{Username}:{Password}";
            var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.Encoding.ASCII.GetBytes(authenticationString));
            requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString);

            var response = await client.SendAsync(requestMessage);
            var contents = await response.Content.ReadAsStringAsync();

            Console.WriteLine(contents);
        }
    }
}
```

{% endtab %}

{% tab title="Golang" %}

```go
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
)

func main() {
	const Username = "USERNAME"
	const Password = "PASSWORD"

	payload := map[string]interface{}{
		"source":       "amazon_product",
		"query":        "B07FZ8S74R",
		"geo_location": "90210",
		"parse":        true,
	}

	jsonValue, _ := json.Marshal(payload)

	client := &http.Client{}
	request, _ := http.NewRequest("POST",
		"https://realtime.oxylabs.io/v1/queries",
		bytes.NewBuffer(jsonValue),
	)

	request.SetBasicAuth(Username, Password)
	request.Header.Set("Content-Type", "application/json")
	response, _ := client.Do(request)

	responseText, _ := ioutil.ReadAll(response.Body)
	fmt.Println(string(responseText))
}
```

{% endtab %}

{% tab title="HTTP" %}

```http
https://realtime.oxylabs.io/v1/queries?source=amazon_product&query=B07FZ8S74R&geo_location=90210&parse=true&access_token=12345abcde
```

{% endtab %}

{% tab title="Java" %}

```java
package org.example;

import okhttp3.*;
import org.json.JSONObject;
import java.util.concurrent.TimeUnit;

public class Main implements Runnable {
    private static final String AUTHORIZATION_HEADER = "Authorization";
    public static final String USERNAME = "USERNAME";
    public static final String PASSWORD = "PASSWORD";

    public void run() {
        JSONObject jsonObject = new JSONObject();
        jsonObject.put("source", "amazon_product");
        jsonObject.put("query", "B07FZ8S74R");
        jsonObject.put("geo_location", "90210");
        jsonObject.put("parse", true);

        Authenticator authenticator = (route, response) -> {
            String credential = Credentials.basic(USERNAME, PASSWORD);
            return response
                    .request()
                    .newBuilder()
                    .header(AUTHORIZATION_HEADER, credential)
                    .build();
        };

        var client = new OkHttpClient.Builder()
                .authenticator(authenticator)
                .readTimeout(180, TimeUnit.SECONDS)
                .build();

        var mediaType = MediaType.parse("application/json; charset=utf-8");
        var body = RequestBody.create(jsonObject.toString(), mediaType);
        var request = new Request.Builder()
                .url("https://realtime.oxylabs.io/v1/queries")
                .post(body)
                .build();

        try (var response = client.newCall(request).execute()) {
            if (response.body() != null) {
                try (var responseBody = response.body()) {
                    System.out.println(responseBody.string());
                }
            }
        } catch (Exception exception) {
            System.out.println("Error: " + exception.getMessage());
        }

        System.exit(0);
    }

    public static void main(String[] args) {
        new Thread(new Main()).start();
    }
}
```

{% endtab %}

{% tab title="JSON" %}

```json
{
    "source": "amazon_product",
    "query": "B07FZ8S74R",
    "geo_location": "90210",
    "parse": true
}
```

{% endtab %}
{% endtabs %}

We use synchronous [**Realtime**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/realtime) integration method in our examples. If you would like to use [**Proxy Endpoint**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/proxy-endpoint) or asynchronous [**Push-Pull**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/push-pull) integration, refer to the [**integration methods**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods) section.

### Forming a request

1. Pick your integration method: synchronous ([**Realtime**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/realtime), [**Proxy Endpoint**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/proxy-endpoint)) or asynchronous  ([**Push-Pull**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/push-pull)).
2. When forming a request, include the following elements:

{% tabs %}
{% tab title="Realtime" %}

1. **Endpoint:**

```
POST https://realtime.oxylabs.io/v1/queries
```

2. **Username and password** ([**HTTP authentication**](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication))**.** Create API user credentials either during your [**trial sign-up or product purchase**](https://dashboard.oxylabs.io/en/registration?productToBuy=SCRAPI_SERP)**.**

```
USERNAME:PASSWORD
```

{% hint style="warning" %}
If you need more than one API user for your account, please contact our [**customer support**](mailto:support@oxylabs.io) or message our 24/7 live chat support.
{% endhint %}

3. **Content-type**. When submitting jobs, always add this header:

```
Content-Type: application/json
```

4. **Payload:**
   1. `source` - This parameter sets the scraper that will be used to process your request.
   2. `URL` or `query` - Provide the `URL` or `query` for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.
   3. [**Additional parameters**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/features): Optionally, you can include additional parameters such as `geo_location`, `user_agent_type`, `parse`, `render` and more to customize your scraping request.

```json
{
    "source": "amazon_product",
    "query": "B07FZ8S74R",
    "geo_location": "90210",
    "parse": true
}
```

{% endtab %}

{% tab title="Proxy Endpoint" %}

1. **Endpoint:**

```
GET realtime.oxylabs.io:60000
```

2. **Ignore certificates.** In `cURL`, it's `-k` or `--insecure`.
3. **Username and password** ([**HTTP authentication**](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication))**.** Create API user credentials either during your [**trial sign-up or product purchase**](https://dashboard.oxylabs.io/en/registration?productToBuy=SCRAPI_SERP)**.**

```
USERNAME:PASSWORD
```

{% hint style="warning" %}
If you need more than one API user for your account, please contact our [**customer support**](mailto:support@oxylabs.io) or message our 24/7 live chat support.
{% endhint %}

4. **Payload:**
   1. `URL` - Provide the `URL` for the page you want to scrape.
   2. [**Additional parameters**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/features): Optionally, you can include additional parameters such as `geo_location`, `user_agent_type`, `parse`, and send them as headers.

```bash
curl -k -x realtime.oxylabs.io:60000 \
-U "USERNAME:PASSWORD" \
'https://www.amazon.com/dp/b07fz8s74r' \
-H "x-oxylabs-geo-location: 90210" \
-H "x-oxylabs-parse: 1"
```

{% endtab %}

{% tab title="Push-Pull" %}

1. **Endpoint:**

```
POST https://data.oxylabs.io/v1/queries
```

2. **Username and password** ([**HTTP authentication**](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication))**.** Create API user credentials either during your [**trial sign-up or product purchase**](https://dashboard.oxylabs.io/en/registration?productToBuy=SCRAPI_SERP)**.**

```
USERNAME:PASSWORD
```

{% hint style="warning" %}
If you need more than one API user for your account, please contact our [**customer support**](mailto:support@oxylabs.io) or message our 24/7 live chat support.
{% endhint %}

3. **Content-type**. When submitting jobs, always add this header:

```
Content-Type: application/json
```

4. **Payload:**
   1. `source` - This parameter sets the scraper that will be used to process your request.
   2. `URL` or `query` - Provide the `URL` or `query` for the type of page you want to scrape. Refer to the table below and the corresponding target sub-pages for detailed guidance on when to use each parameter.
   3. [**Additional parameters**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/features): Optionally, you can include additional parameters such as `geo_location`, `user_agent_type`, `parse`, and more to customize your scraping request.

```json
{
    "source": "amazon_product",
    "query": "B07FZ8S74R",
    "geo_location": "90210",
    "parse": true
}
```

5. Upon submitting a request, you will promptly receive a JSON response containing all job details, including job parameters, job ID, and URLs for downloading job results:

```
GET https://data.oxylabs.io/v1/queries/{job_id}/results
```

```
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=raw
```

```
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=parsed
```

```
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=pn
```

{% endtab %}
{% endtabs %}

<table><thead><tr><th width="217">Target</th><th width="246">Source (Scraping URL)</th><th>Source (Using Query)</th></tr></thead><tbody><tr><td><a href="../scraping-solutions/web-scraper-api/targets/amazon"><strong>Amazon</strong></a></td><td><code>amazon</code></td><td><p><code>amazon_product,</code></p><p><code>amazon_search</code>,</p><p><code>amazon_pricing</code>,</p><p><code>amazon_sellers,</code></p><p><code>amazon_bestsellers,</code></p><p><code>amazon_questions</code></p></td></tr><tr><td><a href="../scraping-solutions/web-scraper-api/targets/google"><strong>Google Shopping</strong></a></td><td><code>google</code></td><td><p><code>google_shopping_product</code>,</p><p><code>google_shopping_search</code>,</p><p><code>google_shopping_pricing</code></p></td></tr><tr><td><a href="../scraping-solutions/web-scraper-api/targets/walmart"><strong>Walmart</strong></a><br><a href="../scraping-solutions/web-scraper-api/targets/north-american-e-commerce/bestbuy"><strong>Best Buy</strong></a><br><a href="../scraping-solutions/web-scraper-api/targets/etsy"><strong>Etsy</strong></a><br><a href="../scraping-solutions/web-scraper-api/targets/north-american-e-commerce/target"><strong>Target</strong></a><br><a href="../scraping-solutions/web-scraper-api/targets/generic-target"><strong>Other Websites</strong></a></td><td><code>universal</code></td><td>Using <code>query</code> parameter is not supported</td></tr></tbody></table>

## Other websites

### Getting started

Scrape any URL with our `universal` source. You can also add [**additional parameters**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/features).&#x20;

### Request sample

{% tabs %}
{% tab title="cURL" %}

```shell
curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "universal",
        "url": "https://sandbox.oxylabs.io/"
    }'
```

{% endtab %}

{% tab title="Python" %}

```python
import requests
from pprint import pprint


# Structure payload.
payload = {
    'source': 'universal',
    'url': 'https://sandbox.oxylabs.io/',
    # 'render': 'html', # If page type requires
}

# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'), # Your credentials go here
    json=payload,
)

# Instead of response with job status and results url, this will return the
# JSON response with results.
pprint(response.json())
```

{% endtab %}

{% tab title="Node.js" %}

```javascript
const https = require("https");

const username = "USERNAME";
const password = "PASSWORD";
const body = {
    source: "universal",
    url: "https://sandbox.oxylabs.io/",
    // 'render': 'html' // If page type requires
};

const options = {
    hostname: "realtime.oxylabs.io",
    path: "/v1/queries",
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        Authorization: "Basic " + Buffer.from(`${username}:${password}`).toString("base64"),
    },
};

const request = https.request(options, (response) => {
    let data = "";

    response.on("data", (chunk) => {
        data += chunk;
    });

    response.on("end", () => {
        const responseData = JSON.parse(data);
        console.log(JSON.stringify(responseData, null, 2));
    });
});

request.on("error", (error) => {
    console.error("Error:", error);
});

request.write(JSON.stringify(body));
request.end();
```

{% endtab %}

{% tab title="HTTP" %}

```http
# The whole string you submit has to be URL-encoded.

https://realtime.oxylabs.io/v1/queries?source=universal&url=https%3A%2F%2Fsandbox.oxylabs.io%2F&access_token=12345abcde
```

{% endtab %}

{% tab title="PHP" %}

```php
<?php

$params = array(
    'source' => 'universal',
    'url' => 'https://sandbox.oxylabs.io/',
    // 'render' => 'html' // If page type requires
);

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "USERNAME" . ":" . "PASSWORD");

$headers = array();
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);
```

{% endtab %}

{% tab title="Golang" %}

```go
package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
)

func main() {
	const Username = "USERNAME"
	const Password = "PASSWORD"

	payload := map[string]interface{}{
		"source": "universal",
		"url":    "https://sandbox.oxylabs.io/",
		// "render": "html" // If page type requires
	}

	jsonValue, _ := json.Marshal(payload)

	client := &http.Client{}
	request, _ := http.NewRequest("POST",
		"https://realtime.oxylabs.io/v1/queries",
		bytes.NewBuffer(jsonValue),
	)

	request.SetBasicAuth(Username, Password)
	response, _ := client.Do(request)

	responseText, _ := ioutil.ReadAll(response.Body)
	fmt.Println(string(responseText))
}
```

{% endtab %}

{% tab title="C#" %}

```csharp
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Net.Http.Json;
using System.Threading.Tasks;

namespace OxyApi
{
    class Program
    {
        static async Task Main()
        {
            const string Username = "USERNAME";
            const string Password = "PASSWORD";

            var parameters = new {
                source = "universal",
                url = "https://sandbox.oxylabs.io/",
                // render = "html", // If page type requires
            };

            var client = new HttpClient();

            Uri baseUri = new Uri("https://realtime.oxylabs.io");
            client.BaseAddress = baseUri;

            var requestMessage = new HttpRequestMessage(HttpMethod.Post, "/v1/queries");
            requestMessage.Content = JsonContent.Create(parameters);

            var authenticationString = $"{Username}:{Password}";
            var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.ASCIIEncoding.UTF8.GetBytes(authenticationString));
            requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString);

            var response = await client.SendAsync(requestMessage);
            var contents = await response.Content.ReadAsStringAsync();

            Console.WriteLine(contents);
        }
    }
}
```

{% endtab %}

{% tab title="Java" %}

```java
package org.example;

import okhttp3.*;
import org.json.JSONObject;
import java.util.concurrent.TimeUnit;

public class Main implements Runnable {
    private static final String AUTHORIZATION_HEADER = "Authorization";
    public static final String USERNAME = "USERNAME";
    public static final String PASSWORD = "PASSWORD";

    public void run() {
        JSONObject jsonObject = new JSONObject();
        jsonObject.put("source", "universal");
        jsonObject.put("url", "https://sandbox.oxylabs.io/");

        Authenticator authenticator = (route, response) -> {
            String credential = Credentials.basic(USERNAME, PASSWORD);
            return response
                    .request()
                    .newBuilder()
                    .header(AUTHORIZATION_HEADER, credential)
                    .build();
        };

        var client = new OkHttpClient.Builder()
                .authenticator(authenticator)
                .readTimeout(180, TimeUnit.SECONDS)
                .build();

        var mediaType = MediaType.parse("application/json; charset=utf-8");
        var body = RequestBody.create(jsonObject.toString(), mediaType);
        var request = new Request.Builder()
                .url("https://realtime.oxylabs.io/v1/queries")
                .post(body)
                .build();

        try (var response = client.newCall(request).execute()) {
            if (response.body() != null) {
                try (var responseBody = response.body()) {
                    System.out.println(responseBody.string());
                }
            }
        } catch (Exception exception) {
            System.out.println("Error: " + exception.getMessage());
        }

        System.exit(0);
    }

    public static void main(String[] args) {
        new Thread(new Main()).start();
    }
}
```

{% endtab %}

{% tab title="JSON" %}

```json
{
    "source": "universal", 
    "url": "https://sandbox.oxylabs.io/"
}
```

{% endtab %}
{% endtabs %}

We use synchronous [**Realtime**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/realtime) integration method in our examples. If you would like to use [**Proxy Endpoint**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/proxy-endpoint) or asynchronous [**Push-Pull**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/push-pull) integration, refer to the [**integration methods**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods) section.

### Forming a request

1. Pick your integration method: synchronous ([**Realtime**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/realtime), [**Proxy Endpoint**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/proxy-endpoint)) or asynchronous  ([**Push-Pull**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/integration-methods/push-pull)).
2. When forming a request, include the following elements:

{% tabs %}
{% tab title="Realtime" %}

1. **Endpoint:**

```
POST https://realtime.oxylabs.io/v1/queries
```

2. **Username and password** ([**HTTP authentication**](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication))**.** Create API user credentials either during your [**trial sign-up or product purchase**](https://dashboard.oxylabs.io/en/registration?productToBuy=SCRAPI_SERP)**.**

```
USERNAME:PASSWORD
```

{% hint style="warning" %}
If you need more than one API user for your account, please contact our [**customer support**](mailto:support@oxylabs.io) or message our 24/7 live chat support.
{% endhint %}

3. **Content-type**. When submitting jobs, always add this header:

```
Content-Type: application/json
```

4. **Payload**.

   1. `source` - This parameter sets the scraper that will be used to process your request.
   2. `URL` - Provide the `URL` of the target you want to scrape, for example:
      1. **Real Estate**: Idealista, Redfin, Zillow, Zoopla
      2. **Travel**: Airbnb, Agoda, Booking, TripAdvisor
      3. **Automotive**: Crunchbase, ZoomInfo, AngelList, Product Hunt
      4. **Company data**: Netflix, SoundCloud, YouTube, IMDb
      5. **Entertainment**: AutoEurope, Autotrader, RockAuto, Halfords
      6. **Any other**.
   3. [**Additional parameters**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/features): Optionally, you can include additional parameters such as `geo_location`, `user_agent_type`, and more to customize your scraping request.

   <pre class="language-json" data-overflow="wrap"><code class="lang-json">{
       "source": "universal",
       "url": "https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/"
   }
   </code></pre>

{% endtab %}

{% tab title="Proxy Endpoint" %}

1. **Endpoint:**

```
GET realtime.oxylabs.io:60000
```

2. **Ignore certificates.** In `cURL`, it's `-k` or `--insecure`.
3. **Username and password** ([**HTTP authentication**](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication))**.** Create API user credentials either during your [**trial sign-up or product purchase**](https://dashboard.oxylabs.io/en/registration?productToBuy=SCRAPI_SERP)**.**

```
USERNAME:PASSWORD
```

{% hint style="warning" %}
If you need more than one API user for your account, please contact our [**customer support**](mailto:support@oxylabs.io) or message our 24/7 live chat support.
{% endhint %}

4. **Payload:**
   1. `URL` - Provide the `URL` for the page you want to scrape, for example:
      1. **Real Estate**: Idealista, Redfin, Zillow, Zoopla
      2. **Travel**: Airbnb, Agoda, Booking, TripAdvisor
      3. **Automotive**: Crunchbase, ZoomInfo, AngelList, Product Hunt
      4. **Company data**: Netflix, SoundCloud, YouTube, IMDb
      5. **Entertainment**: AutoEurope, Autotrader, RockAuto, Halfords
      6. **Any other**.
   2. [**Additional parameters**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/features): Optionally, you can include additional parameters such as `geo_location`, `user_agent_type`, and send them as headers.

{% code overflow="wrap" %}

```bash
curl -k -x realtime.oxylabs.io:60000 \
-U "USERNAME:PASSWORD" \
'https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/' 
```

{% endcode %}
{% endtab %}

{% tab title="Push-Pull" %}

1. **Endpoint:**

```
POST https://data.oxylabs.io/v1/queries
```

2. **Username and password** ([**HTTP authentication**](https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication))**.** Create API user credentials either during your [**trial sign-up or product purchase**](https://dashboard.oxylabs.io/en/registration?productToBuy=SCRAPI_SERP)**.**

```
USERNAME:PASSWORD
```

{% hint style="warning" %}
If you need more than one API user for your account, please contact our [**customer support**](mailto:support@oxylabs.io) or message our 24/7 live chat support.
{% endhint %}

3. **Content-type**. When submitting jobs, always add this header:

```
Content-Type: application/json
```

4. **Payload**.

   1. `source` - This parameter sets the scraper that will be used to process your request.
   2. `URL` - Provide the `URL` of the target you want to scrape, for example:
      1. **Real Estate**: Idealista, Redfin, Zillow, Zoopla
      2. **Travel**: Airbnb, Agoda, Booking, TripAdvisor
      3. **Automotive**: Crunchbase, ZoomInfo, AngelList, Product Hunt
      4. **Company data**: Netflix, SoundCloud, YouTube, IMDb
      5. **Entertainment**: AutoEurope, Autotrader, RockAuto, Halfords
      6. **Any other**.
   3. [**Additional parameters**](https://developers.oxylabs.io/~/changes/1231/scraping-solutions/web-scraper-api/features): Optionally, you can include additional parameters such as `geo_location`, `user_agent_type`, and more to customize your scraping request.

   <pre class="language-json" data-overflow="wrap"><code class="lang-json">{
       "source": "universal",
       "url": "https://www.zillow.com/homedetails/10066-Cielo-Dr-Beverly-Hills-CA-90210/243990393_zpid/"
   }
   </code></pre>
5. Upon submitting a request, you will promptly receive a JSON response containing all job details, including job parameters, job ID, and URLs for downloading job results:

```
GET https://data.oxylabs.io/v1/queries/{job_id}/results
```

```
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=raw
```

```
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=parsed
```

```
GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=pn
```

{% endtab %}
{% endtabs %}

{% hint style="info" %}
If you need any assistance in making your first request, feel free to contact us via the 24/7 available live chat.
{% endhint %}

{% hint style="info" %}
*All information herein is provided on an “as is” basis and for informational purposes only. We make no representation and disclaim all liability with respect to your use of any information contained on this page. Before engaging in scraping activities of any kind you should consult your legal advisors and carefully read the particular website’s terms of service or receive a scraping license.*
{% endhint %}