JavaScript rendering

If the page you wish to scrape requires loading JavaScript to dynamically load all required data into the DOM, instead of setting up and using a headless browser by yourself, you can include the "X-Oxylabs-Render: html" header with your requests. All requests with this header included will be fully rendered, and all data will be stored in an HTML file or PNG screenshot (depending on the passed parameter).

JavaScript rendering takes more time to scrape the page. When using JavaScript rendering, set client side timeout value to 180 seconds.

To ensure lowest traffic consumption, our system does not load unnecessary assets during page rendering.

This parameter has two available values:

  • html (HTML of a rendered page)

  • png (raw bytes that can be saved as PNG)

Code examples

curl -k -v -x unblock.oxylabs.io:60000 \
-U "user:pass1" "https://ip.oxylabs.io/location" \
-H "X-Oxylabs-Render: html"

Scraping a website HTML

In this example, we will render the YouTube home page and scrape the page content. Normally Youtube homepage would look like this if Web Unblocker is used without Javascript rendering:

Adding the "X-Oxylabs-Render: html" header, as provided in the examples below, will enable Javascript rendering and return an HTML of a rendered page:

curl -k -v -x unblock.oxylabs.io:60000 \
-U "user:pass1" "https://youtube.com" \
-H "X-Oxylabs-Render: html"

The HTML file opened in a browser should look like this:

Getting a screenshot of a fully rendered page

To get a screenshot in PNG format instead of page HTML, it is required to provide the "X-Oxylabs-Render: png" header.

curl -k -v -x unblock.oxylabs.io:60000 \
-U "user:pass1" "https://youtube.com" \
-H "X-Oxylabs-Render: png" > rendered_page.png