> For the complete documentation index, see [llms.txt](https://developers.oxylabs.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developers.oxylabs.io/api-targets/cn/ren-he-yu-ming.md). # 任何域名 了解我们专门的网页爬虫API抓取源，适用于 [电子商务网站](/api-targets/cn/dian-zi-shang-wu.md), [搜索引擎](/api-targets/cn/sou-suo-yin-qing.md), [LLMs 和 AI](/api-targets/cn/llm-he-ai.md), [视频数据](/api-targets/cn/shi-pin-he-she-jiao-mei-ti.md), [房地产](/api-targets/cn/fang-di-chan.md) 平台，或使用我们的 `通用` 源，并使用下面的指南。它接受 URL 以及 [附加参数](#additional). ## 请求示例 在此示例中，API 将获取一个电子商务产品页面。 {% tabs %} {% tab title="cURL" %} ```shell curl 'https://realtime.oxylabs.io/v1/queries' \\ --user 'USERNAME:PASSWORD' \\ -H 'Content-Type: application/json' \\ -d '{ "source": "universal", "url": "https://sandbox.oxylabs.io/products/1" }' ``` {% endtab %} {% tab title="Python" %} ```python import requests from pprint import pprint # Structure payload. payload = { 'source': 'universal', 'url': 'https://sandbox.oxylabs.io/products/1', } # Get response. response = requests.request( 'POST', 'https://realtime.oxylabs.io/v1/queries', auth=('USERNAME', 'PASSWORD'), json=payload, ) # Instead of response with job status and results url, this will return the # JSON response with the result. pprint(response.json()) ``` {% endtab %} {% tab title="Node.js" %} ```javascript const https = require("https"); const username = "USERNAME"; const password = "PASSWORD"; const body = { source: "universal", url: "https://sandbox.oxylabs.io/products/1", }; const options = { hostname: "realtime.oxylabs.io", path: "/v1/queries", method: "POST", headers: { "Content-Type": "application/json", Authorization: "Basic " + Buffer.from(`${username}:${password}`).toString("base64"), }, }; const request = https.request(options, (response) => { let data = ""; response.on("data", (chunk) => { data += chunk; }); response.on("end", () => { const responseData = JSON.parse(data); console.log(JSON.stringify(responseData, null, 2)); }); }); request.on("error", (error) => { console.error("Error:", error); }); request.write(JSON.stringify(body)); request.end(); ``` {% endtab %} {% tab title="HTTP" %} ```http # The whole string you submit has to be URL-encoded. https://realtime.oxylabs.io/v1/queries?source=universal&url=https%3A%2F%2Fsandbox.oxylabs.io%2Fproducts%2F1&access_token=12345abcde ``` {% endtab %} {% tab title="PHP" %} ```php 'universal', 'url' => 'https://sandbox.oxylabs.io/products/1', ); $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries"); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params)); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_USERPWD, "USERNAME" . ":" . "PASSWORD"); $headers = array(); $headers[] = "Content-Type: application/json"; curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); $result = curl_exec($ch); echo $result; if (curl_errno($ch)) { echo 'Error:' . curl_error($ch); } curl_close($ch); ``` {% endtab %} {% tab title="Golang" %} ```go package main import ( "bytes" "encoding/json" "fmt" "io/ioutil" "net/http" ) func main() { const Username = "USERNAME" const Password = "PASSWORD" payload := map[string]interface{}{ "source": "universal", "url": "https://sandbox.oxylabs.io/products/1", } jsonValue, _ := json.Marshal(payload) client := &http.Client{} request, _ := http.NewRequest("POST", "https://realtime.oxylabs.io/v1/queries", bytes.NewBuffer(jsonValue), ) request.SetBasicAuth(Username, Password) response, _ := client.Do(request) responseText, _ := ioutil.ReadAll(response.Body) fmt.Println(string(responseText)) } ``` {% endtab %} {% tab title="C#" %} ```csharp using System; using System.Collections.Generic; using System.Net.Http; using System.Net.Http.Json; using System.Threading.Tasks; namespace OxyApi { class Program { static async Task Main() { const string Username = "USERNAME"; const string Password = "PASSWORD"; var parameters = new { source = "universal", url = "https://sandbox.oxylabs.io/products/1" }; var client = new HttpClient(); Uri baseUri = new Uri("https://realtime.oxylabs.io"); client.BaseAddress = baseUri; var requestMessage = new HttpRequestMessage(HttpMethod.Post, "/v1/queries"); requestMessage.Content = JsonContent.Create(parameters); var authenticationString = $"{Username}:{Password}"; var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.ASCIIEncoding.UTF8.GetBytes(authenticationString)); requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString); var response = await client.SendAsync(requestMessage); var contents = await response.Content.ReadAsStringAsync(); Console.WriteLine(contents); } } } ``` {% endtab %} {% tab title="Java" %} ```java package org.example; import okhttp3.*; import org.json.JSONObject; import java.util.concurrent.TimeUnit; public class Main implements Runnable { private static final String AUTHORIZATION_HEADER = "Authorization"; public static final String USERNAME = "USERNAME"; public static final String PASSWORD = "PASSWORD"; public void run() { JSONObject jsonObject = new JSONObject(); jsonObject.put("source", "universal"); jsonObject.put("url", "https://sandbox.oxylabs.io/products/1"); Authenticator authenticator = (route, response) -> { String credential = Credentials.basic(USERNAME, PASSWORD); return response .request() .newBuilder() .header(AUTHORIZATION_HEADER, credential) .build(); }; var client = new OkHttpClient.Builder() .authenticator(authenticator) .readTimeout(180, TimeUnit.SECONDS) .build(); var mediaType = MediaType.parse("application/json; charset=utf-8"); var body = RequestBody.create(jsonObject.toString(), mediaType); var request = new Request.Builder() .url("https://realtime.oxylabs.io/v1/queries") .post(body) .build(); try (var response = client.newCall(request).execute()) { if (response.body() != null) { try (var responseBody = response.body()) { System.out.println(responseBody.string()); } } } catch (Exception exception) { System.out.println("Error: " + exception.getMessage()); } System.exit(0); } public static void main(String[] args) { new Thread(new Main()).start(); } } ``` {% endtab %} {% tab title="JSON" %} ```json { "source": "universal", "url": "https://sandbox.oxylabs.io/products/1" } ``` {% endtab %} {% endtabs %}
输出示例 ```json { "results": [ { "content": " CONTENT ", "created_at": "2024-07-01 11:35:14", "updated_at": "2024-07-01 11:35:15", "page": 1, "url": "https://sandbox.oxylabs.io/products/1", "job_id": "7213505428280329217", "status_code": 200 } ] } ```
我们在示例中使用同步 [**Realtime**](/products/cn/web-scraper-api/integration-methods/realtime.md) 集成方法。如果你想使用 [**Proxy Endpoint**](/products/cn/web-scraper-api/integration-methods/proxy-endpoint.md) 或异步 [**Push-Pull**](/products/cn/web-scraper-api/integration-methods/push-pull.md) 集成，请参考 [**集成方法**](/products/cn/web-scraper-api/integration-methods.md) 部分。 ## 请求参数值 ### 通用
参数说明默认值
source设置爬虫。通用
url任意页面的直接 URL（链接）。-
callback_url指向你的回调端点的 URL。 更多信息.-
\- 必填参数 ### 附加 这些是我们的参数 [**功能**](/products/cn/web-scraper-api/features.md).
参数说明默认值
geo_location将代理的地理位置设置为要检索数据的地区。支持的位置见 此处.-
render当设置为时启用 JavaScript 渲染 html. 更多信息. 注意：如果你观察到成功率较低或返回空内容，请尝试添加此参数。-
browser_instructions定义你自己的浏览器指令，这些指令会在渲染 JavaScript 时执行。 更多信息.-
parse当设置为时返回解析后的数据 true，前提是提交 URL 对应的页面类型存在专用解析器。false
parsing_instructions定义你自己的解析和数据转换逻辑，这些逻辑将在 HTML 抓取结果上执行。了解更多： 解析指令示例.-
context:
headers		传递你自己的 headers。了解更多 此处.-
context:
Cookie		传递你自己的 Cookie。了解更多 此处.-
context:
session_id		如果你想在多个请求中使用同一个代理，可以使用此参数。只需将你的会话设置为任意字符串，我们就会为该 ID 分配一个代理，并最多保留 10 分钟。之后，如果你使用相同的会话 ID 发起另一个请求，将会为该特定会话 ID 分配一个新的代理。-
context:
http_method		将其设置为 post 如果你想发起一个 POST 通过电商爬虫 API 向目标 URL 发起请求。了解更多 此处.get
user_agent_type设备类型和浏览器。完整列表可见于 此处.desktop
context:
content		Base64 编码的 POST 请求正文。仅在以下情况有用： http_method 设置为 post.-
content_encoding如果你正在下载图片，请添加此参数。了解更多 此处.base64
context:
follow_redirects		设为 true 以启用爬虫跟随重定向。默认情况下，重定向最多跟随 10 个链接，并将整个链路视为一个抓取任务。true
context:
successful_status_codes		定义一个自定义 HTTP 响应码（或多个），我们应将其视为抓取成功并向你返回内容。如果你希望我们返回 503 错误页，或其他非标准情况，这可能很有用。-
**所有参数** 在此示例中，包含了所有可用参数（尽管并不总是必要，或在同一请求中兼容），以便你了解请求格式。 {% code fullWidth="false" %} ```json { "source": "universal", "url": "https://example.com", "user_agent_type": "desktop", "geo_location": "美国", "parse": true, "context": [ { "key": "headers", "value": { "Content-Type": "application/octet-stream", "Custom-Header-Name": "自定义请求头内容" } }, { "key": "Cookie", "value": [ { "key": "NID", "value": "1234567890" }, { "key": "1P JAR", "value": "0987654321" }] }, { "key": "follow_redirects", "value": true }, { "key": "http_method", "value": "get" }, { "key": "content", "value": "YmFzZTY0RW5jb2RlZFBPU1RCb2R5" }, { "key": "successful_status_codes", "value": [808, 909] }] } ``` {% endcode %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://developers.oxylabs.io/api-targets/cn/ren-he-yu-ming.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.