任意域名

查找适用于任何公开网站的专用和通用 Oxylabs 网页爬虫API 指南。

探索我们专为以下内容提供的网页爬虫API抓取源： 电商网站, 搜索引擎, LLM 与 AI, 视频数据, 房地产 平台，或使用我们的 universal 源，使用以下指南。它接受 URL 以及 附加参数.

请求示例

在此示例中，API 将获取一个电商产品页面。

cURL

curl 'https://realtime.oxylabs.io/v1/queries' \\
--user 'USERNAME:PASSWORD' \\
-H 'Content-Type: application/json' \\
-d '{
        "source": "universal",
        "url": "https://sandbox.oxylabs.io/products/1"
    }'

Python

import requests
from pprint import pprint


# 结构化载荷。
payload = {
    'source': 'universal',
    'url': 'https://sandbox.oxylabs.io/products/1',
}

# 获取响应。
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'),
    json=payload,
)

# 不会返回包含作业状态和结果 url 的响应，而是返回
# 包含结果的 JSON 响应。
pprint(response.json())

Node.js

const https = require("https");

const username = "USERNAME";
const password = "PASSWORD";
const body = {
    source: "universal",
    url: "https://sandbox.oxylabs.io/products/1",
};

const options = {
    hostname: "realtime.oxylabs.io",
    path: "/v1/queries",
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        Authorization:
            "Basic " + Buffer.from(`${username}:${password}`).toString("base64"),
    },
};

const request = https.request(options, (response) => {
    let data = "";

    response.on("data", (chunk) => {
        data += chunk;
    });

    response.on("end", () => {
        const responseData = JSON.parse(data);
        console.log(JSON.stringify(responseData, null, 2));
    });
});

request.on("error", (error) => {
    console.error("Error:", error);
});

request.write(JSON.stringify(body));
request.end();

HTTP

# The whole string you submit has to be URL-encoded.

https://realtime.oxylabs.io/v1/queries?source=universal&url=https%3A%2F%2Fsandbox.oxylabs.io%2Fproducts%2F1&access_token=12345abcde

PHP

<?php

$params = array(
    'source' => 'universal',
    'url' => 'https://sandbox.oxylabs.io/products/1',
);

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "USERNAME" . ":" . "PASSWORD");

$headers = array();
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);

Golang

package main

import (
	"bytes"
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
)

func main() {
	const Username = "USERNAME"
	const Password = "PASSWORD"

	payload := map[string]interface{}{
		"source": "universal",
		"url":    "https://sandbox.oxylabs.io/products/1",
	}

	jsonValue, _ := json.Marshal(payload)

	client := &http.Client{}
	request, _ := http.NewRequest("POST",
		"https://realtime.oxylabs.io/v1/queries",
		bytes.NewBuffer(jsonValue),
	)

	request.SetBasicAuth(Username, Password)
	response, _ := client.Do(request)

	responseText, _ := ioutil.ReadAll(response.Body)
	fmt.Println(string(responseText))
}

C#

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Net.Http.Json;
using System.Threading.Tasks;

namespace OxyApi
{
    class Program
    {
        static async Task Main()
        {
            const string Username = "USERNAME";
            const string Password = "PASSWORD";

            var parameters = new {
                source = "universal",
                url = "https://sandbox.oxylabs.io/products/1"
            };

            var client = new HttpClient();

            Uri baseUri = new Uri("https://realtime.oxylabs.io");
            client.BaseAddress = baseUri;

            var requestMessage = new HttpRequestMessage(HttpMethod.Post, "/v1/queries");
            requestMessage.Content = JsonContent.Create(parameters);

            var authenticationString = $"{Username}:{Password}";
            var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.ASCIIEncoding.UTF8.GetBytes(authenticationString));
            requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString);

            var response = await client.SendAsync(requestMessage);
            var contents = await response.Content.ReadAsStringAsync();

            Console.WriteLine(contents);
        }
    }
}

Java

package org.example;

import okhttp3.*;
import org.json.JSONObject;
import java.util.concurrent.TimeUnit;

public class Main implements Runnable {
    private static final String AUTHORIZATION_HEADER = "Authorization";
    public static final String USERNAME = "USERNAME";
    public static final String PASSWORD = "PASSWORD";

    public void run() {
        JSONObject jsonObject = new JSONObject();
        jsonObject.put("source", "universal");
        jsonObject.put("url", "https://sandbox.oxylabs.io/products/1");

        Authenticator authenticator = (route, response) => {
            String credential = Credentials.basic(USERNAME, PASSWORD);
            return response
                    .request()
                    .newBuilder()
                    .header(AUTHORIZATION_HEADER, credential)
                    .build();
        };

        var client = new OkHttpClient.Builder()
                .authenticator(authenticator)
                .readTimeout(180, TimeUnit.SECONDS)
                .build();

        var mediaType = MediaType.parse("application/json; charset=utf-8");
        var body = RequestBody.create(jsonObject.toString(), mediaType);
        var request = new Request.Builder()
                .url("https://realtime.oxylabs.io/v1/queries")
                .post(body)
                .build();

        try (var response = client.newCall(request).execute()) {
            if (response.body() != null) {
                try (var responseBody = response.body()) {
                    System.out.println(responseBody.string());
                }
            }
        } catch (Exception exception) {
            System.out.println("Error: " + exception.getMessage());
        }

        System.exit(0);
    }

    public static void main(String[] args) {
        new Thread(new Main()).start();
    }
}

JSON

{
    "source": "universal", 
    "url": "https://sandbox.oxylabs.io/products/1"
}

输出示例

{
    "results": [
        {
            "content": "<!DOCTYPE html><html lang=\"en\">
            CONTENT
            </html>",
            "created_at": "2024-07-01 11:35:14",
            "updated_at": "2024-07-01 11:35:15",
            "page": 1,
            "url": "https://sandbox.oxylabs.io/products/1",
            "job_id": "7213505428280329217",
            "status_code": 200
        }
    ]
}

我们在示例中使用同步 Realtime 集成方法。如果您想使用 Proxy Endpoint 或异步 Push-Pull 集成，请参阅 集成方法 部分。

请求参数值

通用

参数	描述	默认值
source	设置抓取器。	universal
url	任意页面的直接 URL（链接）。	-
callback_url	回调端点的 URL。 更多信息.	-

- 必填参数

附加

以下是我们的 功能的参数.

参数	描述	默认值
geo_location	设置代理的地理位置以获取数据。查找支持的位置 此处.	-
render	设置为时启用 JavaScript 渲染 html. 更多信息. 注意：如果您发现成功率较低或获取到空内容，请尝试添加此参数。	-
browser_instructions	定义在渲染 JavaScript 时执行的自定义浏览器指令。 更多信息.	-
解析	设置为以下值时返回解析后的数据： true，前提是所提交 URL 的页面类型有专用解析器。	false
parsing_instructions	定义将在 HTML 抓取结果上执行的自定义解析和数据转换逻辑。了解更多： 解析指令示例.	-
context: headers	传递您自己的请求头。了解更多 此处.	-
context: Cookie	传递您自己的 Cookie。了解更多 此处.	-
context: session_id	如果您希望在多个请求中使用同一代理，可以使用此参数。只需将会话设置为任意字符串，我们将为此 ID 分配一个代理，并保留最长 10 分钟。之后，如果您使用相同的会话 ID 再次发起请求，将为该会话 ID 分配一个新代理。	-
context: http_method	将其设置为 post 如果您想要向 POST 通过电商爬虫 API 向您的目标 URL 发起请求。了解更多 此处.	get
user_agent_type	设备类型和浏览器。完整列表可在 此处.	desktop
context: content	经过 Base64 编码的 POST 请求正文。仅当 http_method 设置为 post.	-
content_encoding	如果您正在下载图片，请添加此参数。了解更多 此处.	base64
context: follow_redirects	设置为 true 以启用爬虫跟随重定向。默认情况下，最多跟随 10 个链接的重定向，并将整个链视为一个抓取任务。	true
context: successful_status_codes	定义一个或多个自定义 HTTP 响应代码，出现这些代码时我们将视为抓取成功并向您返回内容。如果您希望我们返回 503 错误页面，或在其他非标准情况下，这可能会很有用。	-

所有参数

在此示例中，包含了所有可用参数（尽管它们并非总是必需，也不一定能在同一请求中兼容），以便您了解如何设置请求格式。

{
    "source": "universal", 
    "url": "https://example.com", 
    "user_agent_type": "desktop",
    "geo_location": "United States",
    "parse": true,
    "context": [
        {
            "key": "headers", 
            "value": {
                "Content-Type": "application/octet-stream", 
                "Custom-Header-Name": "custom header content"
            }
        }, 
        {
            "key": "Cookie", 
            "value": [
                {
                    "key": "NID", 
                    "value": "1234567890"
                },
                {
                    "key": "1P JAR",
                    "value": "0987654321"
                }]
        },
        {
            "key": "follow_redirects",
            "value": true
        },
        {
            "key": "http_method", "value": "get"
        },
        {
            "key": "content",
            "value": "YmFzZTY0RW5jb2RlZFBPU1RCb2R5"
        },
        {
            "key": "successful_status_codes",
            "value": [808, 909]
        }]
}