Headers and cookies

Scraper APIs maximizes website unblocking efficiency by using headers and cookies predefined by our team.

Custom headers and cookies are available only for scraping requests with universal and universal_ecommerce sources.

Custom headers

If you require the use of your own headers, you can do so by adding the context:force_headers=true into your requests. Scraper APIs will send your custom headers together with the predefined headers set and then use it to access the website.

This feature allows you to set the values of some of the standard headers like Accept-Language, as well as custom headers tailored to specific targets. However, certain custom headers are always rejected; you can find them in this list.

Custom cookies

Starting on May 10th, 2024, our system will automatically reject any cookies sent by the client to optimize success rates, unless indicated otherwise.

Change preview (Before May 10th):

To facilitate a smooth transition, we offer the option to preview this change. You can assess its impact by setting the value of the context:force_cookies parameter to false to disregard your cookies and see if you need to make any necessary adjustments beforehand.

Code example

{
    "context": [
        {
            "key": "force_cookies",
            "value": false
        },
        {
            "key": "cookies",
            "value": [
                {
                    "key": "NID", 
                    "value": "1234567890"
                }
            ]
        },
    ],
    "source": "universal",
    "url": "https://example.com",
}

Change implementation (After May 10th):

In the event that you need to include your own cookies, you can achieve this by adding context:force_cookies parameter and set it to true when submitting a scraping job. Scraper APIs will send your custom cookies together with the predefined cookies set and then use it to access the website.

Code example

{
    "context": [
        {
            "key": "force_cookies",
            "value": true
        },
        {
            "key": "cookies",
            "value": [
                {
                    "key": "NID", 
                    "value": "1234567890"
                }
            ]
        },
    ],
    "source": "universal",
    "url": "https://example.com",
}