> For the complete documentation index, see [llms.txt](https://developers.oxylabs.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developers.oxylabs.io/integrations/proxy-integrations/scrapy.md). # Scrapy [Scrapy](https://scrapy.org/) is an open-source Python framework designed for large-scale web scraping and data extraction. By default, Scrapy routes requests directly from your local network. Follow the steps below to integrate the following Oxylabs proxies: * [**Residential & Mobile Proxies**](#residential-and-mobile-proxies) * [**Datacenter Proxies**](#datacenter-proxies) * [**Dedicated Datacenter Proxies (Enterprise)**](#dedicated-datacenter-proxies-enterprise) * [**Dedicated Datacenter Proxies (Self-service)**](#dedicated-datacenter-proxies-self-service) * [**ISP Proxies**](#isp-proxies) * [**Dedicated ISP Proxies**](#dedicated-isp-proxies) {% stepper %} {% step %} ### Choose integration method Scrapy supports two primary methods for routing requests through proxies: [**Request Meta Parameter**](#method-1-request-meta-parameter) (per-request configuration) and [**Custom Downloader Middleware**](#method-2-custom-downloader-middleware-recommended) (project-wide configuration). #### Method 1: Request Meta Parameter Pass proxy authentication details directly into individual `scrapy.Request` instances via the `meta` dictionary. This method is recommended for simple spiders or when different requests require specific proxy configurations. ```python import scrapy class OxylabsSpider(scrapy.Spider): name = "oxylabs_spider" def start_requests(self): url = "https://ip.oxylabs.io/" proxy = "http://username:password@host:port" yield scrapy.Request( url=url, callback=self.parse, meta={"proxy": proxy} ) def parse(self, response): ip_address = response.css("pre::text").get() self.logger.info(f"Connected via Oxylabs Proxy. Active IP: {ip_address}") ``` #### Method 2: Custom Downloader Middleware (Recommended) To apply proxies automatically across all requests without altering the spider code, implement a custom Scrapy Downloader Middleware. Open your Scrapy project's `middlewares.py` and add the `OxylabsProxyMiddleware` class: ```python class OxylabsProxyMiddleware: @classmethod def from_crawler(cls, crawler): return cls(crawler.settings) def __init__(self, settings): self.username = settings.get("OXYLABS_USERNAME") self.password = settings.get("OXYLABS_PASSWORD") self.endpoint = settings.get("OXYLABS_ENDPOINT") def process_request(self, request, spider): host = f"http://{self.username}:{self.password}@{self.endpoint}" request.meta["proxy"] = host ``` Then, register the middleware and define your proxy credentials in `settings.py`: ```python # Oxylabs Credentials OXYLABS_USERNAME = "username" OXYLABS_PASSWORD = "password" OXYLABS_ENDPOINT = "host:port" # Enable Downloader Middleware DOWNLOADER_MIDDLEWARES = { "myproject.middlewares.OxylabsProxyMiddleware": 100, "scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110, } ``` {% endstep %} {% step %} ### Configure your proxy Adjust the proxy details and Oxylabs proxy user’s credentials (created in the [Oxylabs dashboard](https://dashboard.oxylabs.io/en/)) for your chosen proxy type: * [Residential & Mobile Proxies](#residential-and-mobile-proxies) * [Datacenter Proxies](#datacenter-proxies) * [Dedicated Datacenter Proxies (Enterprise)](#dedicated-datacenter-proxies-enterprise) * [Dedicated Datacenter Proxies (Self-service)](#dedicated-datacenter-proxies-self-service) * [ISP Proxies](#isp-proxies) * [Dedicated ISP Proxies](#dedicated-isp-proxies) #### Residential & Mobile Proxies **Host:** `pr.oxylabs.io` **Port:** `7777` **Username:** `customer-username` **Password:** `password` {% hint style="warning" %} **Note:** Don't forget to add `customer-` prefix to your username. {% endhint %} You may also use **additional parameters** with residential and mobile proxies to set the geo-location (country, state, city, and etc.) and control sessions. Check out the [Residential Proxies documentation](/products/proxies/residential-proxies.md) to learn more. The table below provides an overview of available parameters and their example values.
Proxy parameterExample
Country entry node

Host: us-pr.oxylabs.io

Port: 10000

CountryUser: customer-username-cc-US
StateUser: customer-username-st-US_texas
CityUser: customer-username-city-paris
ContinentUser: customer-username-cn-NA
ZIP/Postal codeUser: customer-username-cc-US-postalcode-90210
ASN TargetingUser: customer-username-ASN-21928
Session (up to 1440 minutes)User: customer-username-sessid-qwerty123-sesstime-1440
Sticky session

Host: us-pr.oxylabs.io

Port: 10001

#### Datacenter Proxies **Host:** `dc.oxylabs.io` **Port:** `8001` **Username:** `user-userername` **Password:** `password` {% hint style="warning" %} **Note:** Don't forget to add `user-` prefix to your username. {% endhint %} {% tabs %} {% tab title="Pay-per-IP subscription" %} The port is the sequential number assigned to an IP address from the [proxy list](/products/proxies/datacenter-proxies/proxy-list.md). Thus, port `8001` uses the first IP address from your list. You may also enable [proxy IP rotation](/products/proxies/datacenter-proxies/ip-control.md) by setting the port to `8000`. Furthermore, you can specify the [geo-location](/products/proxies/datacenter-proxies/select-country.md), for example, the US, by setting the **Proxy username** to `user-USERNAME-country-US`. {% endtab %} {% tab title="Pay-per-traffic subscription" %} The port `8001` picks a random IP address but stays consistent during a session. Alternatively, you can [rotate proxy IPs](/products/proxies/datacenter-proxies/ip-control.md) with each request by using port `8000`. If you want to use a specific [geo-location](/products/proxies/datacenter-proxies/select-country.md), for example, the US, set the Proxy username to `user-USERNAME-country-US`. {% endtab %} {% endtabs %} #### Dedicated Datacenter Proxies (Enterprise) **Host:** `1.2.3.4` (IP address from your [proxy list](/products/proxies/dedicated-datacenter-proxies/enterprise/proxy-lists.md)) **Port:** `60000` (HTTP) or `1180` (SOCKS5) **Username:** `username` **Password:** `password` For Enterprise Dedicated Datacenter Proxies (purchased through sales team), you’ll have to choose an IP address from the acquired [proxy list](/products/proxies/dedicated-datacenter-proxies/enterprise/proxy-lists.md). #### Dedicated Datacenter Proxies (Self-service) **Host:** `ddc.oxylabs.io` **Port:** `8001` **Username:** `user-userername` **Password:** `password` {% hint style="warning" %} **Note:** Don't forget to add `user-` prefix to your username. {% endhint %} For Self-Service Dedicated Datacenter Proxies (purchased through Oxylabs dashboard), the port indicates the sequential number of an IP address from the acquired [proxy list](/products/proxies/dedicated-datacenter-proxies/self-service/proxy-list.md). If you want to get a different IP with each request, change the Port to `8000` to enable [proxy rotation](/products/proxies/dedicated-datacenter-proxies/self-service/proxy-rotation.md). #### ISP Proxies **Host:** `isp.oxylabs.io` **Port:** `8001` **Username:** `user-username` **Password:** `password` {% hint style="warning" %} **Note:** Don't forget to add `user-` prefix to your username. {% endhint %} With ISP Proxies, the port is assigned to a specific IP address and its location. Hence, port `8001` will always pick the first IP from your [proxy list](/products/proxies/isp-proxies/proxy-list.md). If you want to get a different IP with each request, change the Port to `8000` to enable [proxy rotation](/products/proxies/isp-proxies/proxy-rotation.md). #### Dedicated ISP Proxies **Host:** `disp.oxylabs.io` **Port:** `8001` **Username:** `user-USERNAME` **Password:** `PASSWORD` {% hint style="warning" %} **Note:** Don't forget to add `user-` prefix to your username. {% endhint %} You will need to choose the port assigned to an IP address from the purchased [proxy list](/products/proxies/dedicated-isp-proxies/self-service/proxy-list.md), which you can find on the Oxylabs dashboard. The `8001` port will pick the first IP address from your list for all requests. Use port `8000` for automatic [proxy IP rotation](/products/proxies/dedicated-isp-proxies/self-service/proxy-rotation.md). {% endstep %} {% step %} ### Execute the spider Run your Scrapy spider from the terminal using the `crawl` command: ```bash scrapy crawl oxylabs_spider ``` If configured correctly, Scrapy will execute requests through the Oxylabs proxy gateway and log the assigned external exit IP address. {% endstep %} {% endstepper %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://developers.oxylabs.io/integrations/proxy-integrations/scrapy.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.