> For the complete documentation index, see [llms.txt](https://developers.oxylabs.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developers.oxylabs.io/integrations/cn/web-unblocker-integrations/scrapy.md).

# Scrapy

按照分步 Python 指南将 Oxylabs 网页解锁器与 Scrapy 集成

集成 [Scrapy](https://www.scrapy.org/) 并使用 [Oxylabs 网页解锁器](https://oxylabs.io/products/web-unblocker) 以改善网页访问、处理 CAPTCHA，并自动执行 JavaScript 渲染。本指南涵盖请求元数据和自定义下载中间件。

{% stepper %}
{% step %}

### 选择集成方式

Scrapy 支持通过网页解锁器路由流量的两种方法：Request Meta 参数（按请求配置）和自定义下载中间件（项目级配置）。

#### **方法 1：请求 Meta 参数**

将网页解锁器身份验证详情直接传入单个 `scrapy.Request` 实例，使用 `meta` 字典。你也可以在请求的网页解锁器功能标头中提供自定义 `请求头` 参数。

```python
import scrapy

class UnblockerSpider(scrapy.Spider):
    name = "unblocker_spider"

    def start_requests(self):
        url = "https://sandbox.oxylabs.io/products"
        proxy_uri = "http://YOUR_USERNAME:YOUR_PASSWORD@unblock.oxylabs.io:60000"
        
        # 可选的网页解锁器自定义标头
        headers = {
            "x-oxylabs-render": "html",           # 启用 JavaScript 渲染
            "X-Oxylabs-Geo-Location": "Germany",  # 设置地理定位
        }

        yield scrapy.Request(
            url=url,
            callback=self.parse,
            headers=headers,
            meta={"proxy": proxy_uri}
        )

    def parse(self, response):
        for product in response.css(".product-card"):
            yield {
                "title": product.css(".title::text").get(),
                "price": product.css(".price-wrapper::text").get(),
            }
```

#### **方法 2：自定义下载中间件（推荐）**

要通过 网页解锁器 全局路由所有蜘蛛请求，而无需修改单个请求方法，请实现一个自定义的 Scrapy 下载中间件。

打开 Scrapy 项目的 `middlewares.py` 文件并添加 `OxylabsWebUnblockerMiddleware` 类：

```python
class OxylabsWebUnblockerMiddleware:
    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings)

    def __init__(self, settings):
        self.username = settings.get("OXYLABS_UNBLOCKER_USER")
        self.password = settings.get("OXYLABS_UNBLOCKER_PASS")
        self.endpoint = settings.get("OXYLABS_UNBLOCKER_ENDPOINT", "unblock.oxylabs.io:60000")

    def process_request(self, request, spider):
        proxy_uri = f"http://{self.username}:{self.password}@{self.endpoint}"
        request.meta["proxy"] = proxy_uri
```

然后注册中间件，并指定你的凭据 `settings.py`:

```python
# 网页解锁器 设置
OXYLABS_UNBLOCKER_USER = "YOUR_USERNAME"
OXYLABS_UNBLOCKER_PASS = "YOUR_PASSWORD"
OXYLABS_UNBLOCKER_ENDPOINT = "unblock.oxylabs.io:60000"

# Enable Downloader Middleware
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.OxylabsWebUnblockerMiddleware": 100,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
}
```

{% endstep %}

{% step %}

### 输入凭据和自定义请求头

网页解锁器为所有请求使用单一的集中入口：

**主机：** `unblock.oxylabs.io`

**端口：** `60000`

**用户名：** `YOUR_USERNAME`

**密码：** `YOUR_PASSWORD`

网页解锁器使用 AI 自动生成浏览器指纹和最优请求头。因此，避免传递标准浏览器请求头（例如自定义 `User-Agent` 字符串），除非需要。要控制网页解锁器的行为，请在请求中传递以下自定义请求头：

<table><thead><tr><th width="219">自定义请求头</th><th>说明</th><th width="141">示例值</th></tr></thead><tbody><tr><td><code>x-oxylabs-render</code></td><td>强制对动态、重 JavaScript 的网站使用无头浏览器渲染。了解更多 <a href="/products/cn/web-unblocker/custom-browser-instructions.md">此处</a>.</td><td><code>html</code></td></tr><tr><td><code>X-Oxylabs-Geo-Location</code></td><td>设置地理定位目标（支持国家名称、州或 ZIP 代码）。了解更多 <a href="/products/cn/web-unblocker/making-requests/geo-location.md">此处</a>.</td><td><code>美国</code></td></tr><tr><td><code>X-Oxylabs-Session-Id</code></td><td>在连续请求之间复用相同的 IP 地址和会话。了解更多 <a href="/products/cn/web-unblocker/making-requests/session.md">此处</a>.</td><td><code>session_abc123</code></td></tr><tr><td><code>x-oxylabs-force-headers</code></td><td>强制网页解锁器将您的自定义请求头透传到目标。</td><td><code>1</code></td></tr><tr><td><code>x-oxylabs-force-Cookie</code></td><td>强制网页解锁器保留并向目标站点发送自定义 Cookie。</td><td><code>1</code></td></tr></tbody></table>
{% endstep %}

{% step %}

### 配置响应超时

启用 JavaScript 渲染 (`x-oxylabs-render: html`), 网页解锁器会使用内部无头浏览器实例来执行页面脚本，并等待动态 DOM 元素加载。

由于浏览器渲染比标准 HTTP GET 请求耗时更长，请在 Scrapy 的下载超时设置中 `settings.py` 以防止请求被丢弃：

```python
DOWNLOAD_TIMEOUT = 180
```

{% endstep %}

{% step %}

### 执行爬虫

从命令行运行你的 Scrapy 爬虫：

```bash
scrapy crawl unblocker_spider
```

网页解锁器会通过其 AI 代理基础设施处理请求，管理浏览器指纹识别和重试，并将最终的 HTML 页面载荷直接返回给你的 Scrapy parse 方法。
{% endstep %}
{% endstepper %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://developers.oxylabs.io/integrations/cn/web-unblocker-integrations/scrapy.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.