Proxy Endpoint
If you have ever used regular proxies for data scraping, integrating the Proxy Endpoint delivery method will be a breeze. All you need to do is use our entry node as a proxy, authorize with Scraper API credentials, and ignore certificates. In
cURL, it's -k or --insecure. Your data will reach you on an open connection.Proxy Endpoint only works with the URL-based data sources, where full URL is provided. Therefore, it only accepts a handful of additional job parameters, which should be sent as headers.
GET realtime.oxylabs.io:60000
Please see a request example below.
cURL
Python
PHP
C#
Golang
Java
Node.js
curl -k -x realtime.oxylabs.io:60000 \
-U "USERNAME:PASSWORD" \
-H "x-oxylabs-user-agent-type: desktop_chrome" \
-H "x-oxylabs-geo-location: Germany" \
'https://www.example.com'
import requests
from pprint import pprint
# Define proxy dict. Don't forget to put your real user and pass here as well.
proxies = {
'http': 'http://YOUR_USERNAME:[email protected]:60000',
}
# To set a specific geo-location, user-agent or to render Javascript
# It is required to send parameters as request headers
headers = {
"x-oxylabs-user-agent-type": "desktop_chrome",
"x-oxylabs-geo-location": "Germany",
#"X-Oxylabs-Render": "html", # Uncomment you want to render JavaScript within the page
}
response = requests.request(
'GET',
'https://www.example.com',
headers = headers, # Passing defined headers
verify=False, # Or accept our certificate.
proxies=proxies,
)
# Print result page to stdout
pprint(response.text)
# Save returned HTML to result.html file
with open('result.html', 'w') as f:
f.write(response.text)
<?php
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "https://www.example.com/");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_PROXY, 'realtime.oxylabs.io:60000');
curl_setopt($ch, CURLOPT_PROXYUSERPWD, "YOUR_USERNAME" . ":" . "YOUR_PASSWORD");
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_SSL_VERIFYHOST, false);
// To set a specific geo-location, user-agent or to render Javascript
// It is required to send parameters as request headers
curl_setopt_array($ch, array(
CURLOPT_HTTPHEADER => array(
'x-oxylabs-user-agent-type: desktop_chrome',
'x-oxylabs-geo-location: Germany',
//'X-Oxylabs-Render: html', // Uncomment you want to render JavaScript within the page
)
));
$result = curl_exec($ch);
echo $result;
if (curl_errno($ch)) {
echo 'Error:' . curl_error($ch);
}
curl_close ($ch);
?>
using System;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
namespace OxyApi
{
class Program
{
static async Task Main(string[] args)
{
var webProxy = new WebProxy
{
Address = new Uri($"http://realtime.oxylabs.io:60000"),
BypassProxyOnLocal = false,
UseDefaultCredentials = false,
Credentials = new NetworkCredential(
userName: "YOUR_USERNAME",
password: "YOUR_PASSWORD"
)
};
var httpClientHandler = new HttpClientHandler
{
Proxy = webProxy,
};
// We recommend accepting our certificate instead of allowing insecure (http) traffic
httpClientHandler.ClientCertificateOptions = ClientCertificateOption.Manual;
httpClientHandler.ServerCertificateCustomValidationCallback =
(httpRequestMessage, cert, cetChain, policyErrors) =>
{
return true;
};
var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);
Uri baseUri = new Uri("https://www.example.com");
client.BaseAddress = baseUri;
var requestMessage = new HttpRequestMessage(HttpMethod.Get, "");
var response = await client.SendAsync(requestMessage);
var contents = await response.Content.ReadAsStringAsync();
Console.WriteLine(contents);
}
}
}
package main
import (
"crypto/tls"
"fmt"
"io/ioutil"
"net/http"
"net/url"
)
func main() {
const Username = "YOUR_USERNAME"
const Password = "YOUR_PASSWORD"
proxyUrl, _ := url.Parse(
fmt.Sprintf(
"http://%s:%[email protected]:60000",
Username,
Password,
),
)
customTransport := &http.Transport{Proxy: http.ProxyURL(proxyUrl)}
// We recommend accepting our certificate instead of allowing insecure (http) traffic
customTransport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
client := &http.Client{Transport: customTransport}
request, _ := http.NewRequest("GET",
"https://www.example.com",
nil,
)
request.Header.Add("x-oxylabs-user-agent-type", "desktop_chrome")
request.Header.Add("x-oxylabs-geo-location", "Germany")
request.SetBasicAuth(Username, Password)
response, _ := client.Do(request)
responseText, _ := ioutil.ReadAll(response.Body)
fmt.Println(string(responseText))
}
package org.example;
import okhttp3.Authenticator;
import okhttp3.Credentials;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import javax.net.ssl.*;
import java.net.InetSocketAddress;
import java.net.Proxy;
public class Main implements Runnable {
private static final String AUTHORIZATION_HEADER = "Proxy-Authorization";
public static final String USERNAME = "YOUR_USERNAME";
public static final String PASSWORD = "YOUR_PASSWORD";
public void run() {
Authenticator authenticator = (route, response) -> {
String credential = Credentials.basic(USERNAME, PASSWORD);
return response
.request()
.newBuilder()
.header(AUTHORIZATION_HEADER, credential)
.build();
};
OkHttpClient.Builder builder = new OkHttpClient.Builder();
// We recommend accepting our certificate instead of allowing insecure (http) traffic
this.disableSSLCertificateChecking(builder);
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("realtime.oxylabs.io", 60000));
var client = builder
.proxy(proxy)
.proxyAuthenticator(authenticator)
.build();
var request = new Request.Builder()
.url("https://www.example.com")
.addHeader("x-oxylabs-user-agent-type", "desktop_chrome")
.addHeader("x-oxylabs-geo-location", "Germany")
.get()
.build();
try (var response = client.newCall(request).execute()) {
assert response.body() != null;
System.out.println(response.body().string());
} catch (Exception exception) {
exception.printStackTrace();
System.exit(1);
}
System.exit(0);
}
private void disableSSLCertificateChecking(OkHttpClient.Builder builder) {
TrustManager[] trustManagers = new TrustManager[]{
new X509TrustManager() {
@Override
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return new java.security.cert.X509Certificate[]{};
}
@Override
public void checkServerTrusted(java.security.cert.X509Certificate[] x509Certificates, String authType) {
}
@Override
public void checkClientTrusted(java.security.cert.X509Certificate[] x509Certificates, String authType) {
}
}
};
try {
HttpsURLConnection.setDefaultHostnameVerifier((s, sslSession) -> true);
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, trustManagers, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sslContext.getSocketFactory());
builder.sslSocketFactory(sslContext.getSocketFactory(), (X509TrustManager) trustManagers[0]);
} catch (Exception exception) {
exception.printStackTrace();
System.exit(1);
}
builder.hostnameVerifier((hostname, session) -> true);
}
public static void main(String[] args) {
new Thread(new Main()).start();
}
}
import fetch from 'node-fetch';
import createHttpsProxyAgent from 'https-proxy-agent'
const username = 'YOUR_USERNAME';
const password = 'YOUR_PASSWORD';
const agent = createHttpsProxyAgent(
`http://${username}:${password}@realtime.oxylabs.io:60000`
);
// We recommend accepting our certificate instead of allowing insecure (http) traffic
process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = 0;
const headers = {
'x-oxylabs-user-agent-type': 'desktop_chrome',
'x-oxylabs-geo-location': 'Germany',
}
const response = await fetch('https://www.example.com', {
method: 'get',
headers: headers,
agent: agent,
});
console.log(await response.text());
Below you will find a sample response from
https://example.com:<!doctype html>
<html>
<head>
<title>Example Domain</title>
<meta charset="utf-8" />
<meta http-equiv="Content-type" content="text/html; charset=utf-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<style type="text/css">
body {
background-color: #f0f0f2;
margin: 0;
padding: 0;
font-family: -apple-system, system-ui, BlinkMacSystemFont, "Segoe UI", "Open Sans", "Helvetica Neue", Helvetica, Arial, sans-serif;
}
div {
width: 600px;
margin: 5em auto;
padding: 2em;
background-color: #fdfdff;
border-radius: 0.5em;
box-shadow: 2px 3px 7px 2px rgba(0,0,0,0.02);
}
a:link, a:visited {
color: #38488f;
text-decoration: none;
}
@media (max-width: 700px) {
div {
margin: 0 auto;
width: auto;
}
}
</style>
</head>
<body>
<div>
<h1>Example Domain</h1>
<p>This domain is for use in illustrative examples in documents. You may use this
domain in literature without prior coordination or asking for permission.</p>
<p><a href="https://www.iana.org/domains/example">More information...</a></p>
</div>
</body>
</html>
When making your request, along with the URL, you can send us some job parameters that we will use while executing your job. The job parameters should be sent in your request headers -- see an example here.
Here is the list of job parameters that you can send with Proxy Endpoint requests:
Parameter | Description |
|---|---|
x-oxylabs-user-agent-type | There is no way to indicate a specific User-Agent, but you can let us know which user-agent type you would like us to use. A list of supported User-Agent types can be found here. |
x-oxylabs-geo-location | In some cases, you may need to indicate the geographical location that the result should be adapted for. This parameter corresponds to the geo_location parameter, described separately in source-level documentation. Accepted values depend on the URL you would like us to scrape. |
x-oxylabs-render |
Last modified 3mo ago