Capturing network requests (Fetch/XHR)

Sometimes it is more convenient to extract the required data from one or more of the Fetch/XHR requests that a browser makes while loading the web page, rather than parsing the HTML.

Web Scraper API supports returning the list of XHR requests made while loading the web page.

Usage

To get a list of XHR requests in the response, you can submit a payload similar to the one below:

{
    "url": "https://example.com",
    "render": "html",
    "xhr": true
}

The response will contain a list of JSON objects, each of which representing a single XHR request.

{
    "results": [
        {
            "content": [
                    {
                    "url": "https://example.pilot.ebu.io/api/fallback/RTE",
                    "method": "GET",
                    "status_code": 200,
                    "response_body": "\"https://pilot-fallback.s3-eu-west-1.amazonaws.com/RTE.jpg\"",
                    "request_headers": {
                        "accept": "*/*",
                        "origin": "https://example.reco.ebu.io",
                        "referer": "https://example.com/",
                        "sec-ch-ua": "\"Chromium\";v=\"124\", \"Microsoft Edge\";v=\"124\", \"Not-A.Brand\";v=\"99\"",
                        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.0.0",
                        "sec-fetch-dest": "empty",
                        "sec-fetch-mode": "cors",
                        "sec-fetch-site": "cross-site",
                        "accept-encoding": "gzip, deflate, br, zstd",
                        "accept-language": "en-US,en;q=0.9",
                        "sec-ch-ua-mobile": "?0",
                        "sec-ch-ua-platform": "\"Windows\""
                        },
                    "request_payload": null,
                    "response_headers": {
                        "age": "491",
                        "via": "1.1 3ee81347c1935256691739f42090cfd8.cloudfront.net (CloudFront)",
                        "date": "Fri, 30 May 2025 05:30:44 GMT",
                        "x-eks": "1",
                        "server": "uvicorn",
                        "x-cache": "Hit from cloudfront",
                        "x-amz-cf-id": "ESUATPCUwy-AJ-8OBgKoF3niz-jD87TIyyNypzOKNQL56VCL33XJGw==",
                        "content-type": "application/json",
                        "x-amz-cf-pop": "LAX54-P3",
                        "cache-control": "max-age=600",
                        "content-length": "59",
                        "access-control-allow-origin": "*"
                    }
                },
                {...more xhr requests...}
            ],
            "type": "xhr"
}

Output key	Description	Type
url	Full URL of the XHR request	string
status_code	HTTP status code of the response	integer
method	HTTP method used for the request	string
request_headers	Key-value pairs of request headers, including cookies	object
response_headers	Key-value pairs of response headers, including cookies	object
request_payload	Payload (only include this key with POST, PUT, PATCH requests)	string
response_body	Response body	object