Scrapy
Integrate Scrapy with Oxylabs Proxies to build fast, asynchronous web crawlers using an easy step-by-step guide.
Scrapy is an open-source Python framework designed for large-scale web scraping and data extraction. By default, Scrapy routes requests directly from your local network.
Follow the steps below to integrate the following Oxylabs proxies:
Choose integration method
Scrapy supports two primary methods for routing requests through proxies: Request Meta Parameter (per-request configuration) and Custom Downloader Middleware (project-wide configuration).
Method 1: Request Meta Parameter
Pass proxy authentication details directly into individual scrapy.Request instances via the meta dictionary. This method is recommended for simple spiders or when different requests require specific proxy configurations.
import scrapy
class OxylabsSpider(scrapy.Spider):
name = "oxylabs_spider"
def start_requests(self):
url = "https://ip.oxylabs.io/"
proxy = "http://username:password@host:port"
yield scrapy.Request(
url=url,
callback=self.parse,
meta={"proxy": proxy}
)
def parse(self, response):
ip_address = response.css("pre::text").get()
self.logger.info(f"Connected via Oxylabs Proxy. Active IP: {ip_address}")Method 2: Custom Downloader Middleware (Recommended)
To apply proxies automatically across all requests without altering the spider code, implement a custom Scrapy Downloader Middleware.
Open your Scrapy project's middlewares.py and add the OxylabsProxyMiddleware class:
class OxylabsProxyMiddleware:
@classmethod
def from_crawler(cls, crawler):
return cls(crawler.settings)
def __init__(self, settings):
self.username = settings.get("OXYLABS_USERNAME")
self.password = settings.get("OXYLABS_PASSWORD")
self.endpoint = settings.get("OXYLABS_ENDPOINT")
def process_request(self, request, spider):
host = f"http://{self.username}:{self.password}@{self.endpoint}"
request.meta["proxy"] = hostThen, register the middleware and define your proxy credentials in settings.py:
# Oxylabs Credentials
OXYLABS_USERNAME = "username"
OXYLABS_PASSWORD = "password"
OXYLABS_ENDPOINT = "host:port"
# Enable Downloader Middleware
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.OxylabsProxyMiddleware": 100,
"scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 110,
}Configure your proxy
Adjust the proxy details and Oxylabs proxy user’s credentials (created in the Oxylabs dashboard) for your chosen proxy type:
Residential & Mobile Proxies
Host: pr.oxylabs.io
Port: 7777
Username: customer-username
Password: password
Note: Don't forget to add customer- prefix to your username.
You may also use additional parameters with residential and mobile proxies to set the geo-location (country, state, city, and etc.) and control sessions. Check out the Residential Proxies documentation to learn more. The table below provides an overview of available parameters and their example values.
Host: us-pr.oxylabs.io
Port: 10000
User: customer-username-cc-US
User: customer-username-st-US_texas
User: customer-username-city-paris
User: customer-username-cn-NA
User: customer-username-cc-US-postalcode-90210
User: customer-username-ASN-21928
Session (up to 1440 minutes)
User: customer-username-sessid-qwerty123-sesstime-1440
Host: us-pr.oxylabs.io
Port: 10001
Datacenter Proxies
Host: dc.oxylabs.io
Port: 8001
Username: user-userername
Password: password
Note: Don't forget to add user- prefix to your username.
The port is the sequential number assigned to an IP address from the proxy list. Thus, port 8001 uses the first IP address from your list. You may also enable proxy IP rotation by setting the port to 8000.
Furthermore, you can specify the geo-location, for example, the US, by setting the Proxy username to user-USERNAME-country-US.
The port 8001 picks a random IP address but stays consistent during a session. Alternatively, you can rotate proxy IPs with each request by using port 8000.
If you want to use a specific geo-location, for example, the US, set the Proxy username to user-USERNAME-country-US.
Dedicated Datacenter Proxies (Enterprise)
Host: 1.2.3.4 (IP address from your proxy list)
Port: 60000 (HTTP) or 1180 (SOCKS5)
Username: username
Password: password
For Enterprise Dedicated Datacenter Proxies (purchased through sales team), you’ll have to choose an IP address from the acquired proxy list.
Dedicated Datacenter Proxies (Self-service)
Host: ddc.oxylabs.io
Port: 8001
Username: user-userername
Password: password
Note: Don't forget to add user- prefix to your username.
For Self-Service Dedicated Datacenter Proxies (purchased through Oxylabs dashboard), the port indicates the sequential number of an IP address from the acquired proxy list. If you want to get a different IP with each request, change the Port to 8000 to enable proxy rotation.
ISP Proxies
Host: isp.oxylabs.io
Port: 8001
Username: user-username
Password: password
Note: Don't forget to add user- prefix to your username.
With ISP Proxies, the port is assigned to a specific IP address and its location. Hence, port 8001 will always pick the first IP from your proxy list.
If you want to get a different IP with each request, change the Port to 8000 to enable proxy rotation.
Dedicated ISP Proxies
Host: disp.oxylabs.io
Port: 8001
Username: user-USERNAME
Password: PASSWORD
Note: Don't forget to add user- prefix to your username.
You will need to choose the port assigned to an IP address from the purchased proxy list, which you can find on the Oxylabs dashboard. The 8001 port will pick the first IP address from your list for all requests. Use port 8000 for automatic proxy IP rotation.
Last updated
Was this helpful?