Scrapy

Follow a step-by-step Python guide to integrate Oxylabs Web Unblocker with Scrapy

Integrate Scrapy with Oxylabs Web Unblocker to improve web access, handle CAPTCHAs, and execute JavaScript rendering automatically. This guide covers Request Metadata and Custom Downloader Middleware.

1

Choose integration method

Scrapy supports two methods for routing traffic through Web Unblocker: Request Meta Parameter (per-request configuration) and Custom Downloader Middleware (project-wide configuration).

Method 1: Request Meta Parameter

Pass Web Unblocker authentication details directly into individual scrapy.Request instances using the meta dictionary. You can also supply custom Web Unblocker feature headers inside the request's headers parameter.

import scrapy

class UnblockerSpider(scrapy.Spider):
    name = "unblocker_spider"

    def start_requests(self):
        url = "https://sandbox.oxylabs.io/products"
        proxy_uri = "http://YOUR_USERNAME:[e-mail]:60000"
        
        # Optional Web Unblocker custom headers
        headers = {
            "x-oxylabs-render": "html",           # Enable JavaScript rendering
            "X-Oxylabs-Geo-Location": "Germany",  # Set geo-targeting
        }

        yield scrapy.Request(
            url=url,
            callback=self.parse,
            headers=headers,
            meta={"proxy": proxy_uri}
        )

    def parse(self, response):
        for product in response.css(".product-card"):
            yield {
                "title": product.css(".title::text").get(),
                "price": product.css(".price-wrapper::text").get(),
            }

Method 2: Custom Downloader Middleware (Recommended)

To route all spider requests through Web Unblocker globally without modifying individual request methods, implement a custom Scrapy Downloader Middleware.

Open your Scrapy project's middlewares.py file and add the OxylabsWebUnblockerMiddleware class:

class OxylabsWebUnblockerMiddleware:
    @classmethod
    def from_crawler(cls, crawler):
        return cls(crawler.settings)

    def __init__(self, settings):
        self.username = settings.get("OXYLABS_UNBLOCKER_USER")
        self.password = settings.get("OXYLABS_UNBLOCKER_PASS")
        self.endpoint = settings.get("OXYLABS_UNBLOCKER_ENDPOINT", "unblock.oxylabs.io:60000")

    def process_request(self, request, spider):
        proxy_uri = f"http://{self.username}:{self.password}@{self.endpoint}"
        request.meta["proxy"] = proxy_uri

Then register the middleware and specify your credentials in settings.py:

# Web Unblocker Settings
OXYLABS_UNBLOCKER_USER = "YOUR_USERNAME"
OXYLABS_UNBLOCKER_PASS = "YOUR_PASSWORD"
OXYLABS_UNBLOCKER_ENDPOINT = "unblock.oxylabs.io:60000"

# Enable Downloader Middleware
DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.OxylabsWebUnblockerMiddleware": 100,
    "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
}

2

Enter credentials and custom headers

Web Unblocker uses a single, centralized entry point for all requests:

Host: unblock.oxylabs.io

Port: 60000

Username: YOUR_USERNAME

Password: YOUR_PASSWORD

Web Unblocker uses AI to generate browser fingerprints and optimal request headers automatically. Hence, avoid passing standard browser headers (such as custom User-Agent strings) unless required. To control Web Unblocker behavior, pass the following custom headers in your requests:

Custom Header	Description	Example value
x-oxylabs-render	Forces headless browser rendering for dynamic, JavaScript-heavy sites. Learn more here.	html
X-Oxylabs-Geo-Location	Sets geographical targeting (supports country names, states, or ZIP codes). Learn more here.	United States
X-Oxylabs-Session-Id	Reuses the same IP address and session across sequential requests. Learn more here.	session_abc123
x-oxylabs-force-headers	Forces Web Unblocker to pass your custom request headers through to the target.	1
x-oxylabs-force-cookies	Forces Web Unblocker to preserve and send custom cookies to the target site.	1

3

Configure response timeouts

When enabling JavaScript rendering (x-oxylabs-render: html), Web Unblocker uses an internal headless browser instance to execute page scripts and wait for dynamic DOM elements to load.

Because browser rendering takes longer than standard HTTP GET requests, adjust Scrapy's download timeout in settings.py to prevent request drops:

DOWNLOAD_TIMEOUT = 180

4

Execute the spider

Run your Scrapy spider from the command line:

scrapy crawl unblocker_spider

Web Unblocker will process the request through its AI proxy infrastructure, manage browser fingerprinting and retries, and return the final HTML page payload directly to your Scrapy parse method.