Ctrlk
Access Oxylabs dashboardContact us
x-twitter
For the complete documentation index, see llms.txt. This page is also available as Markdown.

Browser instructions (Beta)

With Web Unblocker, you can define browser instructions when creating a job for a smooth data collection process.

When using Custom Browser Instructions, you can define your own specific browser instructions that are executed when rendering JavaScript.

Generate Browser instructions automatically in the Web Scraper API Playground on the Oxylabs dashboard. Set up browser actions, test them, and export the generated instructions in JSON format.

How to use it?

To use browser instructions, provide a set of browser_instructions when creating a job.

Let’s say you want to search for the term pizza boxes in a website.

An example job parameters would look as follows:

Step 1. You must provide the x-oxylabs-render: html parameter.

Step 2. Browser instructions should be described in the x-oxylabs-browser_instructions field.

The browser instructions provided as the header value must be JSON-escaped and contain no extra spaces.

The sample browser instructions above specifies that the aim is to enter a search term pizza boxes into a search field, click search button and wait for 5 seconds for content to load.

The scraped result should look as follows:

Scraped HTML should look like this:

Fetching browser resources

We provide a standalone browser instruction for fetching browser resources.

The function is defined here:

Using fetch_resource will result in job returning the first occurrence of a Fetch/XHR resource that matches the format provided instead of the HTML that is being targeted.

Let’s say we want to target a GraphQL resource that is fetched when visiting a product page organically in the browser. We will provide job information as such:

These instructions will result in a result as such:

List of supported browser instructions

General arguments

All the instructions defined below have a consistent set of arguments. The arguments are as follows.

type

  • Type: Enum["click", "input", "scroll", "scroll_to_bottom", "wait", "wait_for_element", "fetch_resource"]

  • Description: Browser instruction type.

timeout_s

  • Type: int

  • Description: How long until action is skipped if not completed in time.

  • Restrictions: 0 < timeout_s <= 60

  • Default value: 5

wait_time_s

  • Type: int

  • Description: How long to wait before executing next action.

  • Restrictions: 0 < wait_time_s <= 60

  • Default value: 0

on_error

  • Type: Enum["error", "skip"]

  • Description: Indicator what to do with instructions in case this instruction fails:

    • "error": Stops the execution of browser instructions.

    • "skip": Continues with the next instruction.

  • Default value: "error"

Example with general arguments

Instructions

click

  • Description: Clicks an element and wait a set count of seconds.

  • Args:

    • type: str = "click"

    • selector: dict

      • type: Enum["xpath", "css", "text"]

      • value: str

  • Example:

input

  • Description: Enters a text into a selected element.

  • Args:

    • type: str = "input"

    • selector: dict

      • type: Enum["xpath", "css", "text"]

      • value: str

    • value: str

    • Example:

scroll

  • Description: Scrolls a set count of pixels.

  • Args:

    • type: str = "scroll"

    • x: int

    • y: int

  • Example:

scroll_to_bottom

  • Description: Scrolls to bottom for a set count of seconds.

  • Args:

    • type: str = "scroll_to_bottom"

  • Example:

wait

  • Description: Waits a set count of seconds.

  • Args:

    • type: str = "wait"

  • Example:

wait_for_element

  • Description: Waits for element to load for a set count of seconds.

  • Args:

    • type: str = "wait_for_element"

    • selector: dict

      • type: Enum["xpath", "css", "text"]

      • value: str

    • Example:

fetch_resource

The fetch_resource instruction must be the final instruction in the browser instructions list; any subsequent instructions will not be executed.

  • Description: Fetches the first occurrence of a Fetch/XHR resource matching the set pattern.

  • Args:

    • type: str = "fetch_resource"

    • filter: str(RegEx expression)

    • on_error: Enum["error", "skip"]

  • Example:

Instructions' validation

Any inconsistency in regards to instruction format will result in a 400 status code and a corresponding error message.

For example, payload as such:

Will result in:

Status codes

See our response codes outlined here.

PreviousJavaScript rendering
NextAgent Skills

Last updated

Was this helpful?