Headers, Cookies, Method

Web Scraper API maximizes website unblocking efficiency by using headers and cookies predefined by our team.

Custom headers

If you require the use of your own headers, you can do so by adding the context:force_headers parameter and setting it to true when submitting a scraping job. Web Scraper API will send your custom headers together with the predefined headers set and then use them to access the website.

Code example

{
    "context": [
        {
            "key": "force_headers",
            "value": true
        },
        {
            "key": "headers",
            "value": {
                "Accept-Language": "en-US,en;q=0.5"
            }
        }
    ]
    "source": "universal",
    "url": "https://example.com",
}

Custom cookies

If you require the use of your own cookies, you can do so by adding the context:force_cookies parameter and setting it to true when submitting a scraping job. Web Scraper API will send your custom cookies together with the predefined cookies set and then use them to access the website.

Code example

{
    "context": [
        {
            "key": "force_cookies",
            "value": true
        },
        {
            "key": "cookies",
            "value": [
                {
                    "key": "NID", 
                    "value": "1234567890"
                }
            ]
        },
    ],
    "source": "universal",
    "url": "https://example.com",
}

HTTP Method

universal source uses the GET HTTP method by default.

If you need to post data to the target website, you must set the HTTP method to POST and provide a Base64-encoded POST request body.

Code example

[
    {
        "key": "http_method",
        "value": "post"
    },
    {
        "key": "content",
        "value": "base64EncodedPOSTBody"
    }
]