Proxy Endpoint

Send and receive data via the Oxylabs Web Scraper API Proxy Endpoint. Access target pages directly through a simple URL-based integration.

If you have ever used regular proxies for data scraping, integrating the Proxy Endpoint delivery method will be a breeze. All you need to do is use our entry node as a proxy, authorize with Web Scraper API credentials, and ignore certificates. In cURL, it's -k or --insecure. Your data will reach you on an open connection.

Proxy Endpoint only works with the URL-based data sources, where full URL is provided. Therefore, it only accepts a handful of additional job parameters, which should be sent as headers.

The Proxy Endpoint is not designed to be driven by headless browsers (e.g., Chromium, PhantomJS, Splash) or their automation libraries (e.g., Playwright, Selenium, Puppeteer).

Endpoint

GET realtime.oxylabs.io:60000

Input

Please see a request example below.

cURL

curl -k -x https://realtime.oxylabs.io:60000 \
-U 'USERNAME:PASSWORD' \
-H 'x-oxylabs-user-agent-type: desktop' \
-H 'x-oxylabs-geo-location: Germany' \
'https://www.example.com'

Python

import requests
from pprint import pprint

# Use your Web Scraper API credentials here.
USERNAME, PASSWORD = 'YOUR_USERNAME', 'YOUR_PASSWORD'

# Define proxy dict.
proxies = {
  'http': f'http://{USERNAME}:{PASSWORD}@realtime.oxylabs.io:60000',
  'https': f'https://{USERNAME}:{PASSWORD}@realtime.oxylabs.io:60000'
}

# To set a specific geo-location, user-agent or to render Javascript
# it is required to send parameters as request headers.
headers = {
    'x-oxylabs-user-agent-type': 'desktop',
    'x-oxylabs-geo-location': 'Germany',
    #'X-Oxylabs-Render': 'html', # Uncomment if you want to render JavaScript within the page.
}

response = requests.request(
    'GET',
    'https://www.example.com',
    headers = headers, # Pass the defined headers.
    verify=False,  # Accept our certificate.
    proxies=proxies,
)

# Print result page to stdout.
pprint(response.text)

# Save returned HTML to 'result.html' file.
with open('result.html', 'w') as f:
    f.write(response.text)

Node.js

import fetch from 'node-fetch';
import { HttpsProxyAgent } from 'https-proxy-agent';

const username = 'YOUR_USERNAME';
const password = 'YOUR_PASSWORD';

const agent = new HttpsProxyAgent(
  `https://${username}:${password}@realtime.oxylabs.io:60000`
);

// The Proxy Endpoint presents its own certificate for the target site, so TLS verification must be disabled
process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = 0;

const headers = {
  'x-oxylabs-user-agent-type': 'desktop',
  'x-oxylabs-geo-location': 'Germany',
}

const response = await fetch('https://www.example.com', {
  method: 'get',
  headers: headers,
  agent: agent,
});

console.log(await response.text());

PHP

<?php
$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, 'https://www.example.com/');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXY, 'https://realtime.oxylabs.io:60000');
curl_setopt($ch, CURLOPT_PROXYUSERPWD, 'YOUR_USERNAME' . ':' . 'YOUR_PASSWORD');
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
// To set a specific geo-location, user-agent or to render Javascript
// it is required to send parameters as request headers.
curl_setopt_array($ch, array(
    CURLOPT_HTTPHEADER  => array(
        'x-oxylabs-user-agent-type: desktop',
        'x-oxylabs-geo-location: Germany',
        //'X-Oxylabs-Render: html', // Uncomment if you want to render JavaScript within the page.
    )
));

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);
?>

Golang

package main

import (
    "crypto/tls"
    "fmt"
    "io/ioutil"
    "net/http"
    "net/url"
)

func main() {
    const Username = "YOUR_USERNAME"
    const Password = "YOUR_PASSWORD"

    proxyUrl, _ := url.Parse(
        fmt.Sprintf(
            "https://%s:[e-mail]:60000",
            Username,
            Password,
        ),
    )
    customTransport := &http.Transport{Proxy: http.ProxyURL(proxyUrl)}

    // The Proxy Endpoint presents its own certificate for the target site, so TLS verification must be disabled
    customTransport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}

    client := &http.Client{Transport: customTransport}
    request, _ := http.NewRequest("GET",
        "https://www.example.com",
        nil,
    )

    request.Header.Add("x-oxylabs-user-agent-type", "desktop")
    request.Header.Add("x-oxylabs-geo-location", "Germany")
    request.SetBasicAuth(Username, Password)
    response, _ := client.Do(request)

    responseText, _ := ioutil.ReadAll(response.Body)
    fmt.Println(string(responseText))
}

C#

using System;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;

namespace OxyApi
{
    class Program
    {
        static async Task Main(string[] args)
        {
            var webProxy = new WebProxy
            {
                Address = new Uri($"https://realtime.oxylabs.io:60000"),
                BypassProxyOnLocal = false,
                UseDefaultCredentials = false,

                Credentials = new NetworkCredential(
                userName: "YOUR_USERNAME",
                password: "YOUR_PASSWORD"
                )
            };

            var httpClientHandler = new HttpClientHandler
            {
                Proxy = webProxy,
            };

            // The Proxy Endpoint presents its own certificate for the target site, so TLS verification must be disabled
            httpClientHandler.ClientCertificateOptions = ClientCertificateOption.Manual;
            httpClientHandler.ServerCertificateCustomValidationCallback =
                (httpRequestMessage, cert, cetChain, policyErrors) =>
                {
                    return true;
                };


            var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);

            client.DefaultRequestHeaders.Add("x-oxylabs-user-agent-type", "desktop");
            client.DefaultRequestHeaders.Add("x-oxylabs-geo-location", "Germany");

            Uri baseUri = new Uri("https://www.example.com");
            client.BaseAddress = baseUri;

            var requestMessage = new HttpRequestMessage(HttpMethod.Get, "");

            var response = await client.SendAsync(requestMessage);
            var contents = await response.Content.ReadAsStringAsync();

            Console.WriteLine(contents);
        }
    }
}

Java

package org.example;

import org.apache.hc.client5.http.auth.AuthScope;
import org.apache.hc.client5.http.auth.CredentialsProvider;
import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.hc.client5.http.config.RequestConfig;
import org.apache.hc.client5.http.impl.auth.CredentialsProviderBuilder;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManagerBuilder;
import org.apache.hc.client5.http.ssl.NoopHostnameVerifier;
import org.apache.hc.client5.http.ssl.SSLConnectionSocketFactoryBuilder;
import org.apache.hc.client5.http.ssl.TrustAllStrategy;
import org.apache.hc.core5.http.HttpHost;
import org.apache.hc.core5.http.io.entity.EntityUtils;
import org.apache.hc.core5.http.message.StatusLine;
import org.apache.hc.core5.ssl.SSLContextBuilder;

import java.util.Arrays;
import java.util.Properties;


public class Main {

    public static void main(final String[] args)throws Exception {
        final CredentialsProvider credsProvider = CredentialsProviderBuilder.create()
                .add(new AuthScope("realtime.oxylabs.io", 60000), "YOUR_USERNAME", "YOUR_PASSWORD".toCharArray())
                .build();
        final HttpHost target = new HttpHost("https", "example.com", 443);
        final HttpHost proxy = new HttpHost("https", "realtime.oxylabs.io", 60000);
        try (final CloseableHttpClient httpclient = HttpClients.custom()
                .setDefaultCredentialsProvider(credsProvider)
                .setProxy(proxy)
                // The Proxy Endpoint presents its own certificate for the target site, so TLS verification must be disabled
                .setConnectionManager(PoolingHttpClientConnectionManagerBuilder.create()
                        .setSSLSocketFactory(SSLConnectionSocketFactoryBuilder.create()
                                .setSslContext(SSLContextBuilder.create()
                                        .loadTrustMaterial(TrustAllStrategy.INSTANCE)
                                        .build())
                                .setHostnameVerifier(NoopHostnameVerifier.INSTANCE)
                                .build())
                        .build())
                .build()) {

            final RequestConfig config = RequestConfig.custom()
                    .build();
            final HttpGet request = new HttpGet("/");
            request.addHeader("x-oxylabs-user-agent-type","desktop");
            request.addHeader("x-oxylabs-geo-location","Germany");
            request.setConfig(config);

            System.out.println("Executing request " + request.getMethod() + " " + request.getUri() +
                    " via " + proxy + " headers: " + Arrays.toString(request.getHeaders()));

            httpclient.execute(target, request, response -> {
                System.out.println("----------------------------------------");
                System.out.println(request + "->" + new StatusLine(response));
                EntityUtils.consume(response.getEntity());
                return null;
            });
        }
    }
}

Output

Below you will find a sample response from https://example.com:

Sample response

<!doctype html>
<html lang="en">
<head>
<title>Example Domain</title>
<link rel="icon" href="data:,">
<meta name="viewport" content="width=device-width, initial-scale=1">
<style>body{background:#eee;width:60vw;margin:15vh auto;font-family:system-ui,sans-serif}h1{font-size:1.5em}div{opacity:0.8}a:link,a:visited{color:#348}</style>
</head>
<body>
<div>
<h1>Example Domain</h1>
<p>This domain is for use in documentation examples without needing permission. Avoid use in operations.</p>
<p>
<a href="https://iana.org/domains/example">Learn more</a>
</p>
</div>
</body>
</html>

Accepted parameters

When making your request, along with the URL, you can send us some job parameters that we will use while executing your job. The job parameters should be sent in your request headers - see an example here.

Here is the list of job parameters that you can send with Proxy Endpoint requests:

Parameter	Description
x-oxylabs-user-agent-type	There is no way to indicate a specific User-Agent, but you can let us know which user-agent type you would like us to use. A list of supported User-Agent types can be found here.
x-oxylabs-geo-location	In some cases, you may need to indicate the geographical location that the result should be adapted for. This parameter corresponds to the geo_location parameter, described separately in source-level documentation. Accepted values depend on the URL you would like us to scrape. Read more here.
x-oxylabs-render	JavaScript execution. Accepted values: html and png. Read more here.
x-oxylabs-parse	Set to true to get structured JSON output instead of raw page content. Only works for URLs covered by a dedicated parser.
x-oxylabs-parser-type	Selects a parser explicitly. Must be sent together with x-oxylabs-parse: true, and must be a parser that supports the URL you are scraping.
x-oxylabs-browser-instructions	Browser instructions as a JSON string. Requires x-oxylabs-render. Read more here.