# JavaScript 渲染
如果您要抓取的页面需要加载 JavaScript 才能将所有必需数据动态加载到 DOM 中，与其自己设置并使用无头浏览器，您可以在请求中包含 `"X-Oxylabs-Render: html"` 头部。所有包含此头部的请求将被完全渲染，所有数据将存储在 HTML 文件或 PNG 截图中（取决于传递的参数）。
{% hint style="info" %}
JavaScript 渲染抓取页面需要更多时间。使用 JavaScript 渲染时，请将客户端超时值设置为 180 秒。
{% endhint %}
{% hint style="warning" %}
为确保最低流量消耗，我们的系统在页面渲染期间不会加载不必要的资源。
{% endhint %}
此参数有两个可用值：
* `html` （渲染页面的 HTML）
* `png` （可保存为 PNG 的原始字节）
#### 代码示例
{% tabs %}
{% tab title="cURL" %}
```shell
curl -k -v -x https://unblock.oxylabs.io:60000 \
-U 'USERNAME:PASSWORD' \
'https://ip.oxylabs.io/location' \
-H 'X-Oxylabs-Render: html'
```
{% endtab %}
{% tab title="Python" %}
```python
import requests
# 在此处使用您的 Web Unblocker 凭据。
USERNAME, PASSWORD = 'YOUR_USERNAME', 'YOUR_PASSWORD'
# 定义代理字典。
proxies = {
'http': f'http://{USERNAME}:{PASSWORD}@unblock.oxylabs.io:60000',
'https': f'https://{USERNAME}:{PASSWORD}@unblock.oxylabs.io:60000',
}
headers = {
'X-Oxylabs-Render': 'html'
}
response = requests.get(
'https://ip.oxylabs.io/location',
verify=False, # 需要忽略证书
proxies=proxies,
headers=headers,
)
# 将结果页面打印到标准输出
print(response.text)
# 将返回的 HTML 保存到 result.html 文件
with open('result.html', 'w') as f:
f.write(response.text)
```
{% endtab %}
{% tab title="Node.js" %}
```javascript
import fetch from 'node-fetch';
import { HttpsProxyAgent } from 'https-proxy-agent';
const username = 'YOUR_USERNAME';
const password = 'YOUR_PASSWORD';
const agent = new HttpsProxyAgent(
`https://${username}:${password}@unblock.oxylabs.io:60000`
);
// 我们建议接受我们的证书，而不是允许不安全（http）流量
process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = 0;
const headers = {
'X-Oxylabs-Render': 'html',
}
const response = await fetch('https://ip.oxylabs.io/location', {
method: 'get',
headers: headers,
agent: agent,
});
console.log(await response.text());
```
{% endtab %}
{% tab title="PHP" %}
```php
array(
'X-Oxylabs-Render: html'
)
));
$result = curl_exec($ch);
echo $result;
if (curl_errno($ch)) {
echo 'Error:' . curl_error($ch);
}
curl_close($ch);
```
{% endtab %}
{% tab title="Golang" %}
```go
package main
import (
"crypto/tls"
"fmt"
"io/ioutil"
"net/http"
"net/url"
)
func main() {
const Username = "YOUR_USERNAME"
const Password = "YOUR_PASSWORD"
proxyUrl, _ := url.Parse(
fmt.Sprintf(
"https://%s:%s@unblock.oxylabs.io:60000",
Username,
Password,
),
)
customTransport := &http.Transport{Proxy: http.ProxyURL(proxyUrl)}
// 我们建议接受我们的证书，而不是允许不安全（http）流量
customTransport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
client := &http.Client{Transport: customTransport}
request, _ := http.NewRequest("GET",
"https://ip.oxylabs.io/location",
nil,
)
// 添加自定义 Cookie
request.Header.Add("X-Oxylabs-Render", "html")
request.SetBasicAuth(Username, Password)
response, _ := client.Do(request)
responseText, _ := ioutil.ReadAll(response.Body)
fmt.Println(string(responseText))
}
```
{% endtab %}
{% tab title="C#" %}
```csharp
using System;
using System.Net;
using System.Net.Http;
using System.Threading.Tasks;
namespace OxyApi
{
class Program
{
static async Task Main(string[] args)
{
var webProxy = new WebProxy
{
Address = new Uri("https://unblock.oxylabs.io:60000"),
BypassProxyOnLocal = false,
UseDefaultCredentials = false,
Credentials = new NetworkCredential(
userName: "YOUR_USERNAME",
password: "YOUR_PASSWORD"
)
};
var httpClientHandler = new HttpClientHandler
{
Proxy = webProxy,
};
// 我们建议接受我们的证书，而不是允许不安全（http）流量
httpClientHandler.ClientCertificateOptions = ClientCertificateOption.Manual;
httpClientHandler.ServerCertificateCustomValidationCallback =
(httpRequestMessage, cert, cetChain, policyErrors) =>
{
return true;
};
var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);
// 添加自定义 Cookie
client.DefaultRequestHeaders.Add("X-Oxylabs-Render", "html");
Uri baseUri = new Uri("https://ip.oxylabs.io/location");
client.BaseAddress = baseUri;
var requestMessage = new HttpRequestMessage(HttpMethod.Get, "");
var response = await client.SendAsync(requestMessage);
var contents = await response.Content.ReadAsStringAsync();
Console.WriteLine(contents);
}
}
}
```
{% endtab %}
{% tab title="Java" %}
```java
package org.example;
import org.apache.hc.client5.http.auth.AuthScope;
import org.apache.hc.client5.http.auth.CredentialsProvider;
import org.apache.hc.client5.http.classic.methods.HttpGet;
import org.apache.hc.client5.http.config.RequestConfig;
import org.apache.hc.client5.http.impl.auth.CredentialsProviderBuilder;
import org.apache.hc.client5.http.impl.classic.CloseableHttpClient;
import org.apache.hc.client5.http.impl.classic.HttpClients;
import org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManagerBuilder;
import org.apache.hc.client5.http.ssl.NoopHostnameVerifier;
import org.apache.hc.client5.http.ssl.SSLConnectionSocketFactoryBuilder;
import org.apache.hc.client5.http.ssl.TrustAllStrategy;
import org.apache.hc.core5.http.HttpHost;
import org.apache.hc.core5.http.io.entity.EntityUtils;
import org.apache.hc.core5.http.message.StatusLine;
import org.apache.hc.core5.ssl.SSLContextBuilder;
import java.util.Arrays;
import java.util.Properties;
public class Main {
public static void main(final String[] args)throws Exception {
final CredentialsProvider credsProvider = CredentialsProviderBuilder.create()
.add(new AuthScope("unblock.oxylabs.io", 60000), "USERNAME", "PASSWORD".toCharArray())
.build();
final HttpHost target = new HttpHost("https", "ip.oxylabs.io", 443);
final HttpHost proxy = new HttpHost("https", "unblock.oxylabs.io", 60000);
try (final CloseableHttpClient httpclient = HttpClients.custom()
.setDefaultCredentialsProvider(credsProvider)
.setProxy(proxy)
// 我们建议接受我们的证书，而不是允许不安全（http）流量
.setConnectionManager(PoolingHttpClientConnectionManagerBuilder.create()
.setSSLSocketFactory(SSLConnectionSocketFactoryBuilder.create()
.setSslContext(SSLContextBuilder.create()
.loadTrustMaterial(TrustAllStrategy.INSTANCE)
.build())
.setHostnameVerifier(NoopHostnameVerifier.INSTANCE)
.build())
.build())
.build()) {
final RequestConfig config = RequestConfig.custom()
.build();
final HttpGet request = new HttpGet("/location");
request.addHeader("X-Oxylabs-Render","html");
request.setConfig(config);
System.out.println("Executing request " + request.getMethod() + " " + request.getUri() +
" via " + proxy + " headers: " + Arrays.toString(request.getHeaders()));
httpclient.execute(target, request, response -> {
System.out.println("----------------------------------------");
System.out.println(request + "->" + new StatusLine(response));
EntityUtils.consume(response.getEntity());
return null;
});
}
}
}
```
{% endtab %}
{% endtabs %}
**抓取网站 HTML**
在此示例中，我们将渲染 YouTube 主页并抓取页面内容。通常如果在不使用 Javascript 渲染的情况下使用 Web Unblocker，YouTube 主页通常如下所示：
未启用 JavaScript 渲染时的 YouTube 页面示例
**获取完全渲染页面的截图**
要以 PNG 格式获取截图而不是页面 HTML，需要提供 `"X-Oxylabs-Render: png"` 头部。
{% tabs %}
{% tab title="cURL" %}
```shell
curl -k -v -x https://unblock.oxylabs.io:60000 \
-U 'USERNAME:PASSWORD' \
'https://youtube.com' \
-H 'X-Oxylabs-Render: png' > rendered_page.png
```
{% endtab %}
{% tab title="Python" %}
```python
import requests
# 在此处使用您的 Web Unblocker 凭据。
USERNAME, PASSWORD = 'YOUR_USERNAME', 'YOUR_PASSWORD'
# 定义代理字典。
proxies = {
'http': f'http://{USERNAME}:{PASSWORD}@unblock.oxylabs.io:60000',
'https': f'https://{USERNAME}:{PASSWORD}@unblock.oxylabs.io:60000',
}
headers = {
'X-Oxylabs-Render': 'png'
}
response = requests.get(
'https://youtube.com',
verify=False, # 需要忽略证书
proxies=proxies,
headers=headers,
)
# 将截图保存为 PNG 文件
with open('rendered_page.png', 'wb') as f:
f.write(response.content)
```
{% endtab %}
{% tab title="Node.js" %}
```javascript
import fs from 'fs';
import fetch from 'node-fetch';
import { HttpsProxyAgent } from 'https-proxy-agent';
const username = 'YOUR_USERNAME';
const password = 'YOUR_PASSWORD';
const agent = new HttpsProxyAgent(
`https://${username}:${password}@unblock.oxylabs.io:60000`
);
// 我们建议接受我们的证书，而不是允许不安全（http）流量
process.env['NODE_TLS_REJECT_UNAUTHORIZED'] = 0;
const headers = {
'X-Oxylabs-Render': 'png',
}
await fetch('https://youtube.com', {
method: 'get',
headers: headers,
agent: agent,
}).then(res =>
res.body.pipe(fs.createWriteStream('./rendered_page.png'))
);
console.log('Image was saved');
```
{% endtab %}
{% tab title="PHP" %}
```php
[
'X-Oxylabs-Render: png'
]
]);
$result = curl_exec($ch);
if (curl_errno($ch)) {
echo 'Error:' . curl_error($ch);
}
curl_close ($ch);
if(file_exists('rendered_page.png')){
unlink('rendered_page.png');
}
$fp = fopen('rendered_page.png','wb');
fwrite($fp, $result);
fclose($fp);
```
{% endtab %}
{% tab title="Golang" %}
```go
package main
import (
"crypto/tls"
"fmt"
"io/ioutil"
"net/http"
"net/url"
)
func main() {
const Username = "YOUR_USERNAME"
const Password = "YOUR_PASSWORD"
proxyUrl, _ := url.Parse(
fmt.Sprintf(
"https://%s:%s@unblock.oxylabs.io:60000",
Username,
Password,
),
)
customTransport := &http.Transport{Proxy: http.ProxyURL(proxyUrl)}
// 我们建议接受我们的证书，而不是允许不安全（http）流量
customTransport.TLSClientConfig = &tls.Config{InsecureSkipVerify: true}
client := &http.Client{Transport: customTransport}
request, _ := http.NewRequest("GET",
"https://youtube.com",
nil,
)
// 添加自定义 Cookie
request.Header.Add("X-Oxylabs-Render", "png")
request.SetBasicAuth(Username, Password)
response, _ := client.Do(request)
responseData, _ := ioutil.ReadAll(response.Body)
response.Body.Close()
ioutil.WriteFile("rendered_page.png", responseData, 0666)
log.Println("Image was saved")
}
```
{% endtab %}
{% tab title="C#" %}
using System;
using System.Net;
using System.Net.Http;
using System.Drawing;
using System.IO;
using System.Threading.Tasks;
namespace OxyApi
{
class Program
{
static async Task Main(string[] args)
{
var webProxy = new WebProxy
{
Address = new Uri("https://unblock.oxylabs.io:60000"),
BypassProxyOnLocal = false,
UseDefaultCredentials = false,
Credentials = new NetworkCredential(
userName: "YOUR_USERNAME",
password: "YOUR_PASSWORD"
)
};
var httpClientHandler = new HttpClientHandler
{
Proxy = webProxy,
};
// 我们建议接受我们的证书，而不是允许不安全（http）流量
httpClientHandler.ClientCertificateOptions = ClientCertificateOption.Manual;
httpClientHandler.ServerCertificateCustomValidationCallback =
(httpRequestMessage, cert, cetChain, policyErrors) =>
{
return true;
};
var client = new HttpClient(handler: httpClientHandler, disposeHandler: true);
// 添加自定义头部
client.DefaultRequestHeaders.Add("X-Oxylabs-Render", "png");
Uri baseUri = new Uri("https://youtube.com");
client.BaseAddress = baseUri;
var requestMessage = new HttpRequestMessage(HttpMethod.Get, "");
var response = await client.SendAsync(requestMessage);
byte[] bytes = await response.Content.ReadAsByteArrayAsync();
using (Image image = Image.FromStream(new MemoryStream(bytes)))
{
image.Save("rendered_page.png");
}
}
}
}
{% endtab %}
{% tab title="Java" %}
```java
package org.example;
import okhttp3.Authenticator;
import okhttp3.Credentials;
import okhttp3.OkHttpClient;
import okhttp3.Request;
import javax.net.ssl.*;
import java.net.InetSocketAddress;
import java.net.Proxy;
import java.util.concurrent.TimeUnit;
import java.io.File;
import org.apache.commons.io.FileUtils;
public class App implements Runnable
{
private static final String AUTHORIZATION_HEADER = "Proxy-Authorization";
public static final String USERNAME = "YOUR_USERNAME";
public static final String PASSWORD = "YOUR_PASSWORD";
public void run() {
Authenticator authenticator = (route, response) -> {
String credential = Credentials.basic(USERNAME, PASSWORD);
return response
.request()
.newBuilder()
.header(AUTHORIZATION_HEADER, credential)
.build();
};
OkHttpClient.Builder builder = new OkHttpClient.Builder()
.readTimeout(180, TimeUnit.SECONDS)
.writeTimeout(180, TimeUnit.SECONDS);
// 我们建议接受我们的证书，而不是允许不安全（http）流量
this.disableSSLCertificateChecking(builder);
Proxy proxy = new Proxy(Proxy.Type.HTTP, new InetSocketAddress("unblock.oxylabs.io", 60000));
OkHttpClient client = builder
.proxy(proxy)
.proxyAuthenticator(authenticator)
.build();
var request = new Request.Builder()
.url("https://youtube.com")
.addHeader("X-Oxylabs-Render", "png")
.get()
.build();
try (var response = client.newCall(request).execute()) {
assert response.body() != null;
okhttp3.ResponseBody body = response.body();
if (body != null) {
byte[] bytes = body.bytes();
FileUtils.writeByteArrayToFile(new File("rendered_page.png"), bytes);
}
} catch (Exception exception) {
exception.printStackTrace();
System.exit(1);
}
System.exit(0);
}
private void disableSSLCertificateChecking(OkHttpClient.Builder builder) {
TrustManager[] trustManagers = new TrustManager[]{
new X509TrustManager() {
@Override
public java.security.cert.X509Certificate[] getAcceptedIssuers() {
return new java.security.cert.X509Certificate[]{};
}
@Override
public void checkServerTrusted(java.security.cert.X509Certificate[] x509Certificates, String authType) {
}
@Override
public void checkClientTrusted(java.security.cert.X509Certificate[] x509Certificates, String authType) {
}
}
};
try {
HttpsURLConnection.setDefaultHostnameVerifier((s, sslSession) -> true);
SSLContext sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, trustManagers, new java.security.SecureRandom());
HttpsURLConnection.setDefaultSSLSocketFactory(sslContext.getSocketFactory());
builder.sslSocketFactory(sslContext.getSocketFactory(), (X509TrustManager) trustManagers[0]);
} catch (Exception exception) {
exception.printStackTrace();
System.exit(1);
}
builder.hostnameVerifier((hostname, session) -> true);
}
public static void main(String[] args) {
new Thread(new App()).start();
}
}
```
{% endtab %}
{% endtabs %}
响应将包含图像的原始字节，可保存为 PNG 格式并按下面示例打开：
YouTube 页面示例（以 PNG 格式的截图）
## 在特定页面强制渲染
为成功抓取，某些特定域的页面类型由于其动态内容需要渲染。我们的系统会自动对这些页面强制渲染，即使用户未明确设置。
{% hint style="warning" %}
请注意，渲染任务比非渲染任务消耗更多流量。
{% endhint %}
我们希望用户在抓取以下页面时对此有充分认识：
{% file src="" %}
这种方法为这些具有挑战性的页面提供了最佳抓取体验，确保数据的准确性和可靠性。
如果您希望禁用渲染，请发送不带值的渲染头
```
X-oxylabs-render:
```