> For the complete documentation index, see [llms.txt](https://developers.oxylabs.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developers.oxylabs.io/integrations/web-unblocker-integrations/scrapy.md).

# Scrapy

Follow a step-by-step Python guide to integrate Oxylabs Web Unblocker with Scrapy

Integrate [Scrapy](https://www.scrapy.org/) with [Oxylabs Web Unblocker](https://oxylabs.io/products/web-unblocker) to improve web access, handle CAPTCHAs, and execute JavaScript rendering automatically. This guide covers Request Metadata and Custom Downloader Middleware.

{% stepper %}
{% step %}

### Choose integration method

Scrapy supports two methods for routing traffic through Web Unblocker: Request Meta Parameter (per-request configuration) and Custom Downloader Middleware (project-wide configuration).

#### **Method 1: Request Meta Parameter**

Pass Web Unblocker authentication details directly into individual `scrapy.Request` instances using the `meta` dictionary. You can also supply custom Web Unblocker feature headers inside the request's `headers` parameter.

```python
import scrapy

class UnblockerSpider(scrapy.Spider):
    name = "unblocker_spider"

    def start_requests(self):
        url = "https://sandbox.oxylabs.io/products"
        proxy_uri = "http://YOUR_USERNAME:YOUR_PASSWORD@unblock.oxylabs.io:60000"
        
        # Optional Web Unblocker custom headers
        headers = {
            "x-oxylabs-render": "html",           # Enable JavaScript rendering
            "X-Oxylabs-Geo-Location": "Germany",  # Set geo-targeting
        }

        yield scrapy.Request(
            url=url,
            callback=self.parse,
            headers=headers,
            meta={"proxy": proxy_uri}
        )

    def parse(self, response):
        for product in response.css(".product-card"):
            yield {
                "title": product.css(".title::text").get(),
                "price": product.css(".price-wrapper::text").get(),
            }
```

#### **Method 2: Custom Downloader Middleware (Recommended)**

To route all spider requests through Web Unblocker globally without modifying individual request methods, implement a custom Scrapy Downloader Middleware.

Open your Scrapy project's `middlewares.py` file and add the `OxylabsWebUnblockerMiddleware` class:

```python
class OxylabsWebUnblockerMiddleware:
    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings)

    def __init__(self, settings):
        self.username = settings.get("OXYLABS_UNBLOCKER_USER")
        self.password = settings.get("OXYLABS_UNBLOCKER_PASS")
        self.endpoint = settings.get("OXYLABS_UNBLOCKER_ENDPOINT", "unblock.oxylabs.io:60000")

    def process_request(self, request, spider):
        proxy_uri = f"http://{self.username}:{self.password}@{self.endpoint}"
        request.meta["proxy"] = proxy_uri
```

Then register the middleware and specify your credentials in `settings.py`:

```python
# Web Unblocker Settings
OXYLABS_UNBLOCKER_USER = "YOUR_USERNAME"
OXYLABS_UNBLOCKER_PASS = "YOUR_PASSWORD"
OXYLABS_UNBLOCKER_ENDPOINT = "unblock.oxylabs.io:60000"

# Enable Downloader Middleware
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.OxylabsWebUnblockerMiddleware": 100,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
}
```

{% endstep %}

{% step %}

### Enter credentials and custom headers

Web Unblocker uses a single, centralized entry point for all requests:

**Host:** `unblock.oxylabs.io`

**Port:** `60000`

**Username:** `YOUR_USERNAME`

**Password:** `YOUR_PASSWORD`

Web Unblocker uses AI to generate browser fingerprints and optimal request headers automatically. Hence, avoid passing standard browser headers (such as custom `User-Agent` strings) unless required. To control Web Unblocker behavior, pass the following custom headers in your requests:

<table><thead><tr><th width="219">Custom Header</th><th>Description</th><th width="141">Example value</th></tr></thead><tbody><tr><td><code>x-oxylabs-render</code></td><td>Forces headless browser rendering for dynamic, JavaScript-heavy sites. Learn more <a href="/products/web-unblocker/custom-browser-instructions.md">here</a>.</td><td><code>html</code></td></tr><tr><td><code>X-Oxylabs-Geo-Location</code></td><td>Sets geographical targeting (supports country names, states, or ZIP codes). Learn more <a href="/products/web-unblocker/making-requests/geo-location.md">here</a>.</td><td><code>United States</code></td></tr><tr><td><code>X-Oxylabs-Session-Id</code></td><td>Reuses the same IP address and session across sequential requests. Learn more <a href="/products/web-unblocker/making-requests/session.md">here</a>.</td><td><code>session_abc123</code></td></tr><tr><td><code>x-oxylabs-force-headers</code></td><td>Forces Web Unblocker to pass your custom request headers through to the target.</td><td><code>1</code></td></tr><tr><td><code>x-oxylabs-force-cookies</code></td><td>Forces Web Unblocker to preserve and send custom cookies to the target site.</td><td><code>1</code></td></tr></tbody></table>
{% endstep %}

{% step %}

### Configure response timeouts

When enabling JavaScript rendering (`x-oxylabs-render: html`), Web Unblocker uses an internal headless browser instance to execute page scripts and wait for dynamic DOM elements to load.

Because browser rendering takes longer than standard HTTP GET requests, adjust Scrapy's download timeout in `settings.py` to prevent request drops:

```python
DOWNLOAD_TIMEOUT = 180
```

{% endstep %}

{% step %}

### Execute the spider

Run your Scrapy spider from the command line:

```bash
scrapy crawl unblocker_spider
```

Web Unblocker will process the request through its AI proxy infrastructure, manage browser fingerprinting and retries, and return the final HTML page payload directly to your Scrapy parse method.
{% endstep %}
{% endstepper %}


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:

```
GET https://developers.oxylabs.io/integrations/web-unblocker-integrations/scrapy.md?ask=<question>&goal=<endgoal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.