# 通用来源 使用我们的爬取其他网站（Home Depot、Idealo、Zillow、 [**Yandex**](https://developers.oxylabs.io/documentation/cn/wang-ye-pa-chong-api-zhi-nan/forming-urls#yandex), [**Baidu**](https://developers.oxylabs.io/documentation/cn/wang-ye-pa-chong-api-zhi-nan/forming-urls#baidu)等）的 `universal` 源。它接受 URL 以及 [**附加参数**](#additional). ## 请求示例 在此示例中，API 将检索一个电商产品页面。 {% tabs %} {% tab title="cURL" %} ```shell curl 'https://realtime.oxylabs.io/v1/queries' \ --user 'USERNAME:PASSWORD' \ -H 'Content-Type: application/json' \ -d '{ "source": "universal", "url": "https://sandbox.oxylabs.io/products/1" }' ``` {% endtab %} {% tab title="Python" %} ```python import requests from pprint import pprint # 组织有效负载。 payload = { 'source': 'universal', 'url': 'https://sandbox.oxylabs.io/products/1', } # 获取响应。 response = requests.request( 'POST', 'https://realtime.oxylabs.io/v1/queries', auth=('USERNAME', 'PASSWORD'), json=payload, ) # 与返回作业状态和结果 URL 相比，这将返回 # 带有结果的 JSON 响应。 pprint(response.json()) ``` {% endtab %} {% tab title="Node.js" %} ```javascript const https = require("https"); const username = "USERNAME"; const password = "PASSWORD"; const body = { source: "universal", url: "https://sandbox.oxylabs.io/products/1", }; const options = { hostname: "realtime.oxylabs.io", path: "/v1/queries", method: "POST", headers: { "Content-Type": "application/json", Authorization: "Basic " + Buffer.from(`${username}:${password}`).toString("base64"), }, }; const request = https.request(options, (response) => { let data = ""; response.on("data", (chunk) => { data += chunk; }); response.on("end", () => { const responseData = JSON.parse(data); console.log(JSON.stringify(responseData, null, 2)); }); }); request.on("error", (error) => { console.error("Error:", error); }); request.write(JSON.stringify(body)); request.end(); ``` {% endtab %} {% tab title="HTTP" %} ```http # 您提交的整个字符串必须进行 URL 编码。 https://realtime.oxylabs.io/v1/queries?source=universal&url=https%3A%2F%2Fsandbox.oxylabs.io%2Fproducts%2F1&access_token=12345abcde ``` {% endtab %} {% tab title="PHP" %} ```php 'universal', 'url' => 'https://sandbox.oxylabs.io/products/1', ); $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries"); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params)); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_USERPWD, "USERNAME" . ":" . "PASSWORD"); $headers = array(); $headers[] = "Content-Type: application/json"; curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); $result = curl_exec($ch); echo $result; if (curl_errno($ch)) { echo 'Error:' . curl_error($ch); } curl_close($ch); ``` {% endtab %} {% tab title="Golang" %} ```go package main import ( "bytes" "encoding/json" "fmt" "io/ioutil" "net/http" ) func main() { const Username = "USERNAME" const Password = "PASSWORD" payload := map[string]interface{}{ "source": "universal", "url": "https://sandbox.oxylabs.io/products/1", } jsonValue, _ := json.Marshal(payload) client := &http.Client{} request, _ := http.NewRequest("POST", "https://realtime.oxylabs.io/v1/queries", bytes.NewBuffer(jsonValue), ) request.SetBasicAuth(Username, Password) response, _ := client.Do(request) responseText, _ := ioutil.ReadAll(response.Body) fmt.Println(string(responseText)) } ``` {% endtab %} {% tab title="C#" %} ```csharp using System; using System.Collections.Generic; using System.Net.Http; using System.Net.Http.Json; using System.Threading.Tasks; namespace OxyApi { class Program { static async Task Main() { const string Username = "USERNAME"; const string Password = "PASSWORD"; var parameters = new { source = "universal", url = "https://sandbox.oxylabs.io/products/1" }; var client = new HttpClient(); Uri baseUri = new Uri("https://realtime.oxylabs.io"); client.BaseAddress = baseUri; var requestMessage = new HttpRequestMessage(HttpMethod.Post, "/v1/queries"); requestMessage.Content = JsonContent.Create(parameters); var authenticationString = $"{Username}:{Password}"; var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.ASCIIEncoding.UTF8.GetBytes(authenticationString)); requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString); var response = await client.SendAsync(requestMessage); var contents = await response.Content.ReadAsStringAsync(); Console.WriteLine(contents); } } } ``` {% endtab %} {% tab title="Java" %} ```java package org.example; import okhttp3.*; import org.json.JSONObject; import java.util.concurrent.TimeUnit; public class Main implements Runnable { private static final String AUTHORIZATION_HEADER = "Authorization"; public static final String USERNAME = "USERNAME"; public static final String PASSWORD = "PASSWORD"; public void run() { JSONObject jsonObject = new JSONObject(); jsonObject.put("source", "universal"); jsonObject.put("url", "https://sandbox.oxylabs.io/products/1"); Authenticator authenticator = (route, response) -> { String credential = Credentials.basic(USERNAME, PASSWORD); return response .request() .newBuilder() .header(AUTHORIZATION_HEADER, credential) .build(); }; var client = new OkHttpClient.Builder() .authenticator(authenticator) .readTimeout(180, TimeUnit.SECONDS) .build(); var mediaType = MediaType.parse("application/json; charset=utf-8"); var body = RequestBody.create(jsonObject.toString(), mediaType); var request = new Request.Builder() .url("https://realtime.oxylabs.io/v1/queries") .post(body) .build(); try (var response = client.newCall(request).execute()) { if (response.body() != null) { try (var responseBody = response.body()) { System.out.println(responseBody.string()); } } } catch (Exception exception) { System.out.println("Error: " + exception.getMessage()); } System.exit(0); } public static void main(String[] args) { new Thread(new Main()).start(); } } ``` {% endtab %} {% tab title="JSON" %} ```json { "source": "universal", "url": "https://sandbox.oxylabs.io/products/1" } ``` {% endtab %} {% endtabs %}
输出示例 ```json { "results": [ { "content": " 内容 ", "created_at": "2024-07-01 11:35:14", "updated_at": "2024-07-01 11:35:15", "page": 1, "url": "https://sandbox.oxylabs.io/products/1", "job_id": "7213505428280329217", "status_code": 200 } ] } ```
我们在示例中使用同步的 [**Realtime**](https://developers.oxylabs.io/documentation/cn/zhua-qu-jie-jue-fang-an/web-scraper-api/integration-methods/realtime) 集成方法。如果您想使用 [**Proxy Endpoint**](https://developers.oxylabs.io/documentation/cn/zhua-qu-jie-jue-fang-an/web-scraper-api/integration-methods/proxy-endpoint) 或异步的 [**Push-Pull**](https://developers.oxylabs.io/documentation/cn/zhua-qu-jie-jue-fang-an/web-scraper-api/integration-methods/push-pull) 集成，请参阅 [**集成方法**](https://developers.oxylabs.io/documentation/cn/zhua-qu-jie-jue-fang-an/web-scraper-api/integration-methods) 部分。 ## 请求参数值 ### 通用
参数描述默认值
source设置爬虫。universal
url指向任意页面的直接 URL（链接）。-
callback_url您的回调端点的 URL。 更多信息.-
\- 必需参数 ### 附加 这些是我们 [**功能**](https://developers.oxylabs.io/documentation/cn/zhua-qu-jie-jue-fang-an/web-scraper-api/features).
参数描述默认值
geo_location设置用于检索数据的代理地理位置。查找支持的地点 这里.-
render设置为时启用 JavaScript 渲染 html. 更多信息. 注意：如果您观察到成功率低或检索到空内容，请尝试添加此参数。-
browser_instructions定义在渲染 JavaScript 时执行的自定义浏览器指令。 更多信息.-
parse设置为时返回解析后的数据 true，只要为提交 URL 的页面类型存在专用解析器。false
parsing_instructions定义将在 HTML 抓取结果上执行的自定义解析和数据转换逻辑。阅读更多： 解析指令示例.-
context:
headers		传递您自己的头。了解更多 这里.-
context:
cookies		传递您自己的 Cookie。了解更多 这里.-
context:
session_id		如果您想对多个请求使用相同代理，可以使用此参数。只需将会话设置为任意字符串，我们将为该 ID 分配代理，并保持该代理最长 10 分钟。之后，如果您使用相同的会话 ID 发起另一个请求，将为该会话 ID 分配新代理。-
context:
http_method		将其设置为 post 如果您想通过电商爬虫 API 对目标 URL 发起 POST 请求。了解更多 这里.get
user_agent_type设备类型和浏览器。完整列表可在此找到 这里.desktop
context:
content		Base64 编码的 POST 请求体。仅在 http_method 设置为时有用 post.-
content_encoding如果您正在下载图片，请添加此参数。了解更多 这里.base64
context:
follow_redirects		设置为 true 以启用爬虫跟随重定向。默认情况下，重定向最多跟随 10 个链接，将整个链视为一个抓取作业。true
context:
successful_status_codes		定义自定义 HTTP 响应代码（或多个），当遇到这些代码时我们应将抓取视为成功并返回内容。这在您希望我们返回 503 错误页面或其他非标准情况时可能有用。-
**所有参数** 在此示例中，包含了所有可用参数（尽管并非在同一请求中总是必要或兼容），以便让您了解如何格式化请求。 {% code fullWidth="false" %} ```json { "source": "universal", "url": "https://example.com", "user_agent_type": "desktop", "geo_location": "United States", "parse": true, "context": [ { "key": "headers", "value": { "Content-Type": "application/octet-stream", "Custom-Header-Name": "custom header content" } }, { "key": "cookies", "value": [ { "key": "NID", "value": "1234567890" }, { "key": "1P JAR", "value": "0987654321" }] }, { "key": "follow_redirects", "value": true }, { "key": "http_method", "value": "get" }, { "key": "content", "value": "YmFzZTY0RW5jb2RlZFBPU1RCb2R5" }, { "key": "successful_status_codes", "value": [808, 909] }] } ``` {% endcode %} --- # Agent Instructions: Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://developers.oxylabs.io/documentation/cn/zhua-qu-jie-jue-fang-an/web-scraper-api/targets/generic-target.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.