Headers, Cookies, POST

Scraper APIs maximizes website unblocking efficiency by using headers and cookies predefined by our team.

Custom headers and cookies are available only for scraping other search engines (excluding Google and Bing) with the universal source.

Custom headers

If you require the use of your own headers, you can do so by adding the context:force_headers parameter and setting it to true when submitting a scraping job. Scraper APIs will send your custom headers together with the predefined headers set and then use them to access the website.

This feature allows you to set the values of some of the standard headers like Accept-Language, as well as custom headers tailored to specific targets. However, certain custom headers are always rejected; you can find them in this list.

Code example

{
    "context": [
        {
            "key": "force_headers",
            "value": true
        },
        {
            "key": "headers",
            "value": {
                "Accept-Language": "en-US,en;q=0.5"
            }
        }
    ]
    "source": "universal",
    "url": "https://example.com",
}

Custom cookies

If you require the use of your own cookies, you can do so by adding the context:force_cookies parameter and setting it to true when submitting a scraping job. Scraper APIs will send your custom cookies together with the predefined cookies set and then use them to access the website.

Code example

{
    "context": [
        {
            "key": "force_cookies",
            "value": true
        },
        {
            "key": "cookies",
            "value": [
                {
                    "key": "NID", 
                    "value": "1234567890"
                }
            ]
        },
    ],
    "source": "universal",
    "url": "https://example.com",
}

HTTP Method

universal source uses the GET HTTP method by default.

If you need to post data to the target website, you must set the HTTP method to POST and provide a Base64-encoded POST request body.

Code example

{
    "context": [
        {
            "key": "http_method",
            "value": "post"
        },
        {
            "key": "content",
            "value": "base64EncodedPOSTBody"
        },
    ],
    "source": "universal",
    "url": "https://example.com",
}