JavaScript Rendering
If the page you wish to scrape requires JavaScript to dynamically load all necessary data into the DOM, you can include a render
parameter in your requests instead of setting up and using a headless browser manually. Requests with this parameter will be fully rendered, and the data will be stored in either an HTML file or a PNG screenshot, depending on the specified parameter.
HTML
Set the render
parameter to html
to get the raw output of the rendered page.
PNG (Screenshot)
Set the render
parameter to png
to get a Base64-encoded screenshot of the rendered page.
Request sample
curl --user "user:pass" \
'https://realtime.oxylabs.io/v1/queries' \
-H "Content-Type: application/json" \
-d '{"source": "universal", "url": "https://www.example.com", "render": "html"}'
JavaScript rendering takes more time to scrape the page. Please set timeout on the client side to 180 seconds if using Realtime or Proxy Endpoint integration methods.
To ensure lowest traffic consumption, our system does not load unnecessary assets during page rendering.
Forcing rendering on specific pages
For successful scraping, some page types of specific domains require rendering due to their dynamic content. Our system automatically enforces rendering for these pages, even if not explicitly set by the user.
Please note that rendered jobs consume more traffic compared to non-rendered jobs.
We want our users to be fully aware of this when scraping the following pages:
This approach provides the best possible scraping experience, ensuring data accuracy and reliability from these challenging pages.
If you wish to disable rendering, you can do so by adding the following parameter to your requests:
"render": ""
Browser instructions
With our Headless Browser you can also execute various browser instructions such as clicking, scrolling, inputting, waiting and more. Read more:
Browser InstructionsLast updated
Was this helpful?