Documentation has been updated: see help center and changelog in one place.
Explore
LogoLogo
Oxylabs dashboardProduct
English
  • Documentation
  • Help center
  • Changelog
English
  • Overview
  • PROXIES
    • Integration Guides
      • Get IP Address for Integrations
      • Residential Proxies Guides
        • AdsPower
        • Android
        • ClonBrowser
        • Dolphin Anty
        • Ghost Browser
        • GoLogin
        • Helium Scraper
        • Incogniton
        • iOS
        • Kameleo
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Multilogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • ParseHub
        • Playwright
        • Puppeteer
        • Selenium
        • SEO Neo
        • SessionBox
        • Shadowrocket
        • Super Proxy
        • SwitchyOmega
        • Ubuntu
        • VMLogin
        • WebHarvy
        • Hidemyacc
      • ISP Proxies Guides
        • AdsPower
        • Android
        • Dolphin Anty
        • FoxyProxy
        • GoLogin
        • Incogniton
        • iOS
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Multilogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • SEO Neo
        • Shadowrocket
        • Sphere
        • Super Proxy
        • SwitchyOmega
        • Ubuntu
        • Hidemyacc
      • Mobile Proxies Guides
        • AdsPower
        • Android
        • ClonBrowser
        • Dolphin Anty
        • Ghost Browser
        • GoLogin
        • Helium Scraper
        • Incogniton
        • iOS
        • Kameleo
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Multilogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • ParseHub
        • Playwright
        • Puppeteer
        • Selenium
        • SEO Neo
        • SessionBox
        • Shadowrocket
        • SwitchyOmega
        • Ubuntu
        • VMLogin
        • WebHarvy
      • Dedicated Datacenter Proxies Guides
        • Enterprise
          • Dolphin Anty
          • FoxyProxy
          • GoLogin
          • Lalicat Browser
          • MoreLogin
          • MuLogin
          • Nstbrowser
          • Octoparse
          • Oxy® Proxy Extension for Chrome
          • SEO Neo
          • Shadowrocket
          • Sphere
          • Super Proxy
          • SwitchyOmega
          • Ubuntu
          • Hidemyacc
        • Self-Service
          • Android
          • Dolphin Anty
          • FoxyProxy
          • GoLogin
          • iOS
          • Lalicat Browser
          • MacOS
          • MoreLogin
          • MuLogin
          • Nstbrowser
          • Octoparse
          • Oxy® Proxy Extension for Chrome
          • SEO Neo
          • Shadowrocket
          • Sphere
          • Super Proxy
          • SwitchyOmega
          • Ubuntu
          • Hidemyacc
      • Datacenter Proxies Guides
        • AdsPower
        • Android
        • Dolphin Anty
        • GoLogin
        • iOS
        • Lalicat Browser
        • MacOS
        • MoreLogin
        • MuLogin
        • Nstbrowser
        • Octoparse
        • Oxy® Proxy Extension for Chrome
        • SEO Neo
        • Shadowrocket
        • Super Proxy
        • SwitchyOmega
        • Ubuntu
        • Hidemyacc
    • Residential Proxies
      • Getting Started
      • Making Requests
        • Entry Node for China
      • Location Settings
        • Country
        • City
        • State
        • Continent
        • ZIP/Postal code
        • Coordinates
        • ASN Targeting
      • Session Control
        • Sticky Proxy Entry Nodes
      • Protocols
      • Whitelisting IPs
        • Requests with Whitelisted IPs
      • Endpoint Generator
      • Restricted Targets
      • Public API
      • Response Codes
    • ISP Proxies
      • Making Requests
      • Proxy List
      • Proxy Rotation
      • Location Settings
      • Protocols
      • Whitelisting IPs
      • Response Codes
      • Restricted Targets
      • Fair usage policy
    • Mobile Proxies
      • Getting Started
      • Making Requests
        • Entry Node for China
      • Location Settings
        • Country
        • City
        • State
        • Continent
        • Coordinates
        • ASN Targeting
      • Session Control
        • Sticky Proxy Entry Nodes
      • Protocols
      • Whitelisting IPs
      • Endpoint Generator
      • Restricted Targets
      • Public API
      • Response Codes
    • Datacenter Proxies
      • Proxy List
      • IP Control
      • Select Country
      • Protocols
      • Whitelisting
      • Response Codes
      • Restricted Targets
      • Fair usage policy
      • Free Datacenter IPs
    • Dedicated Datacenter Proxies
      • Enterprise
        • Getting Started
        • Proxy List
        • Making Requests
        • Protocols
        • Whitelisting IPs
          • Dashboard
          • RESTful
            • Getting Whitelisted IPs List
            • Adding a Whitelisted IP
            • Removing a Whitelisted IP
            • Saving Changes (5min Cooldown)
        • Datacenter Proxy API
        • Proxy Rotator - Optional
        • Response Codes
      • Self-Service
        • Getting Started
        • Making Requests
        • Proxy List
        • Proxy Rotation
        • Location Settings
        • Protocols
        • Whitelisting IPs
        • Response Codes
        • Restricted Targets
        • Fair usage policy
    • Dedicated ISP Proxies
      • Getting Started
      • Proxy List
      • Making Requests
      • Protocols
      • Whitelisting IPs (RESTful)
        • Getting Whitelisted IPs List
        • Adding a Whitelisted IP
        • Removing a Whitelisted IP
        • Saving Changes (5min Cooldown)
      • Proxy API
      • Proxy Rotator - Optional
      • Response Codes
    • (LEGACY) Shared Datacenter Proxies
      • Getting Started
      • Making Requests
      • Select Country
      • Session Control
      • Response Codes
      • Restricted Targets
  • Advanced proxy solutions
    • Web Unblocker
      • Getting Started
      • Making Requests
        • Session
        • Geo-location
        • Headers & Cookies
        • Custom status code
        • POST requests
      • Headless Browser
        • JavaScript rendering
        • Browser instructions (Beta)
          • List of instructions
      • Sample Response
      • Response Codes
      • Rate Limits
      • Migration Guides
        • From Bright Data Web Unlocker
      • Usage Statistics
      • Billing Information
  • Scraper APIs
    • Web Scraper API
      • Integration Methods
        • Realtime
        • Push-Pull
        • Proxy Endpoint
      • Features
        • Localization
          • Proxy Location
          • SERP Localization
          • E-Commerce Localization
          • Domain, Locale, Results Language
        • JS Rendering & Browser Control
          • JavaScript Rendering
          • Browser Instructions
            • List of instructions
        • Result Processing & Storage
          • Dedicated Parsers
          • Custom Parser
            • Getting started
            • Parsing instruction examples
            • List of functions
              • Function examples
          • Download Images
          • Cloud Storage
        • HTTP Context & Job Management
          • Headers, Cookies, Method
          • User Agent Type
          • Client Notes
        • Scheduler
      • Solutions for AI Workflows
        • Model Context Protocol (MCP)
      • Targets
        • Google
          • Search
            • Web Search
            • AI Overviews
            • Image Search
            • News Search
            • Local Search
            • Reverse Image Search
            • Google Suggest
          • Ads Max
          • Shopping
            • Shopping Product
            • Shopping Search
            • Shopping Pricing
          • Trends: Explore
          • Travel: Hotels
          • Lens
          • URL
        • Amazon
          • Product
          • Search
          • Pricing
          • Sellers
          • Best Sellers
          • Reviews
          • Questions & Answers
          • URL
        • YouTube
          • YouTube Scraping Guide for AI
          • YouTube Search
          • YouTube Video Trainability
          • YouTube Metadata
          • YouTube Downloader
          • YouTube Transcript
        • Generic Target
        • Walmart
          • Search
          • Product
        • Ebay
        • Etsy
          • Search
          • Product
        • Bing
          • Search
          • URL
        • North American E-Commerce
          • Best Buy
            • Search
            • Product
          • Target
            • Search
            • Product
            • Category
          • Kroger
            • Product
            • Search
            • URL
          • Bed Bath & Beyond
          • Menards
          • Petco
          • Staples
          • Grainger
          • Instacart
        • European E-Commerce
          • Allegro
            • Search
            • Product
          • Idealo
          • Mediamarkt
          • Cdiscount
        • Asian E-Commerce
          • Alibaba
          • Aliexpress
          • Lazada
          • Rakuten
          • Tokopedia
          • Flipkart
          • Avnet
          • Indiamart
        • Latin American E-Commerce
          • Mercado Livre
          • Magazine Luiza
          • Falabella
          • Dcard
      • Restricted Targets
      • Response Codes
    • OxyCopilot (Beta)
    • Usage and Billing
      • Usage Statistics
      • Traffic and Billing
      • Rate Limits
  • Dashboard
    • Teams
    • Billing Information
      • Accessing Billing Information
      • Managing Payment Methods
      • Updating Billing Information
      • Canceling a Subscription
    • IP Replacement
  • Guides for Scraper APIs
    • Python SDK
    • Go SDK
    • Forming Requests
    • Forming URLs
    • Using Postman
  • Useful links
    • Oxylabs Dashboard
    • Release Notes
    • Network status
    • Open Source Tools
      • Oxy Parser
      • Oxy Mouse
      • Web Scraper API Scheduler
    • Discord Community
    • GitHub
    • Scraping Experts
  • SUPPORT
    • FAQ
    • Have a Question?
Powered by GitBook
On this page
  • HTML processing
  • element_text
  • xpath
  • xpath_one
  • String manipulation
  • amount_from_string
  • amount_range_from_string
  • join
  • regex_find_all
  • regex_search
  • regex_substring
  • Common functions
  • convert_to_*
  • select_nth
  • Math functions
  • average
  • max
  • min
  • product

Was this helpful?

  1. Scraper APIs
  2. Web Scraper API
  3. Features
  4. Result Processing & Storage
  5. Custom Parser
  6. List of functions

Function examples

HTML processing

element_text

Sample HTML

<!DOCTYPE html>
<html>
<body>
    <div id="product">
        <div id="product-description">This is a nice product</div>
        <div id="product-price">    12  3


        </div>
    </div>
</body>
</html>

Extract text from HTML element and strip whitespaces

{
    "price": {
        "_fns": [
            {
                "_fn": "xpath_one",
                "_args": [".//*[@id='product-price']"]
            },
            {
                "_fn": "element_text"
            }
        ]
    }
}
{
    "price": "12  3"
}

Given a string value as an input, do nothing

{
    "price": {
        "_fns": [
            {
                "_fn": "xpath_one",
                "_args": [".//*[@id='product-price']/text()"]
            },
            {
                "_fn": "element_text"
            }
        ]
    }
}
{
    "price": "    12  3\n\n\n        "
}

xpath

Sample HTML

<body>
    <div class="product" id="socks">
        <div class="title">Socks</div>
        <div class="price">123.12</div>
        <div class="description">
            <ul>
                <li class="description-item">Very</li>
                <li class="description-item">Nice</li>
                <li class="description-item">Socks</li>
            </ul>
        </div>
    </div>
</body>

Get all description items

{
    "description_items": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": ["//li[@class='description-item']/text()"]
            }
        ]
    }
}
{
    "description_items": ["Very", "Nice", "Socks"]
}

Get the first description item

{
    "first_description_item": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": ["(//li[@class='description-item'])[1]/text()"]
            }
        ]
    }
}
{
    "first_description_item": [
        "Very"
    ]
}

Check if the description section element exists

{
    "description_section_exists": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": ["boolean(//div[@class='description'])"]
            }
        ]
    }
}
{
    "description_section_exists": true
}

Get price as a number

{
    "price": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": ["number(//div[@class='price'])"]
            }
        ]
    }
}
{
    "description_section_exists": 123.12
}

Multiple expressions to fallback to in case preceding expression fails

{
    "price": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": [
                    "//div[@class='product-price']/text()", <--- this does not find anything
                    "//div[@class='price']/text()" <--- this finds the target price
                ]
            }
        ]
    }
}
{
    "price": [
        "123.12"
    ]
}

XPath | operator to match with multiple expressions

{
    "price_and_title": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": ["//div[@class='price']/text() | //div[@class='title']/text()"]
            }
        ]
    }
}
{
    "price_and_title": [
        "Socks",
        "123.12"
    ]
}

xpath_one

Sample HTML

<body>
    <div class="product" id="socks">
        <div class="title">Socks</div>
        <div class="price">123.12</div>
        <div class="description">
            <ul>
                <li class="description-item">Very</li>
                <li class="description-item">Nice</li>
                <li class="description-item">Socks</li>
            </ul>
        </div>
    </div>
</body>

Return the first match

{
    "first_description_item": {
        "_fns": [
            {
                "_fn": "xpath_one",
                "_args": [".//li/text()"]
            }
        ]
    }
}
{
    "first_description_item": "Very"
}

Using XSLT functions

{
    "price": {
        "_fns": [
            {
                "_fn": "xpath_one",
                "_args": ["number(.//div[@class='price'])"]
            }
        ]
    }
}
{
    "price": 123.12
}

String manipulation

amount_from_string

Sample HTML

<body>
    <div class="product" id="socks">
        <div class="title">Socks</div>
        <div class="price">The price is: 123.12 pesos</div>
    </div>
</body>

Extract amount from string

{
    "price": {
        "_fns": [
            {
                "_fn": "xpath_one",
                "_args": [".//div[@class='price']/text()"]
            },
            {
                "_fn": "amount_from_string"
            }
        ]
    }
}
{
    "price": 123.12
}

amount_range_from_string

Sample HTML

<body>
    <div class="product">
        <div class="price">
            The price is: 123.12 pesos;
            The price is: 345.12 pesos;
            The price is: 678.12 pesos
        </div>
    </div>
</body>

Extract all amounts from string

{
    "prices": {
        "_fns": [
            {
                "_fn": "xpath_one",
                "_args": [".//div[@class='price']/text()"]
            },
            {
                "_fn": "amount_range_from_string"
            }
        ]
    }
}
{    
    "prices": [
        123.12,
        345.12,
        678.12
    ]
}

join

Sample HTML

<body>
    <div class="product">
        <div class="price">
            The price is: 123.12 pesos;
        </div>
        <div class="price">
            The price is: 345.12 pesos;
        </div>
        <div class="price">
            The price is: 678.12 pesos
        </div>
    </div>
</body>

Join an array of strings into a single string

{
    "price_variants": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": [".//div[@class='price']"]
            },
            {  // If we call normalize-space() in first pipeline function, 
               // it will return only the first value.
                "_fn": "xpath",
                "_args": ["normalize-space(text())"]
            },  
            {
                "_fn": "join",
                "_args": ""
            }
        ]
    }
}
{
    "price_variants": "The price is: 123.12 pesos;The price is: 345.12 pesos;The price is: 678.12 pesos"
}

regex_find_all

Sample HTML

<body>
    <div class="product">
        <div class="description">
            [one description]
            [two description]
            [three description]
        </div>
    </div>
</body>

Find all matches between two characters

{
    "descriptions": {
        "_fns": [
            {
                "_fn": "xpath_one",
                "_args": [".//div[@class='description']/text()"]
            },
            {
                "_fn": "regex_find_all",
                "_args": ["\\[(.*)\\]"]
            }
        ]
    }
}
{
    "descriptions": [
        "one description",
        "two description",
        "three description"
    ]
}

regex_search

Sample HTML

<body>
    <div class="product">
        <div class="description">
            [one description]
            [two description]
            [three description]
            {the one i need}
        </div>
    </div>
</body>

Return description between two characters

{
    "description": {
        "_fns": [
            {
                "_fn": "xpath_one",
                "_args": [".//div[@class='description']/text()"]
            },
            {
                "_fn": "regex_search",
                "_args": ["{(.*)}", 1]
            }
        ]
    }
}
{
    "description": "the one i need"
}

regex_substring

Sample HTML

<body>
    <div class="product">
        <div class="description">
            * one description
            * two description
            * three description
            * {this one i would like to get replaced}
        </div>
    </div>
</body>

Replace a part of text with specified value

{
    "descriptions": {
        "_fns": [
            {
                "_fn": "xpath_one",
                "_args": [".//div[@class='description']/text()"]
            },
            {
                "_fn": "regex_substring",
                "_args": ["{this one i would like to get replaced}", "four description"]
            },
            {
                "_fn": "regex_find_all",
                "_args": ["\\*\\s(.*)\n"]
            }
        ]
    }
}
{
    "descriptions": [
        "one description",
        "two description",
        "three description",
        "four description"
    ]
}

Common functions

convert_to_*

Sample HTML

<body>
    <div class="product">
        <div class="price">123</div>
        <div class="price">124</div>
        <div class="price">456</div>
        <div class="price">421</div>
        <div class="price">100</div>
    </div>
</body>

Get the count of price variants

{
    "price_variants": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": [".//div[@class='price']"]
            },
            {
                "_fn": "length"
            }
        ]
    }
}
{
    "price_variants": 5
}

Get the count of price variants in a multi-dimensional array

Sample HTML:

<body>
    <div class="product">
        <property class="colors">
            <option class="color">Red</option>
            <option class="color">Green</option>
            <option class="color">Blue</option>
        </property>
        <property class="sizes">
            <option class="size">S</option>
            <option class="size">M</option>
            <option class="size">L</option>
            <option class="size">XL</option>
        </property>
    </div>
</body>
{
    "number_of_variants": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": [".//property"]
            },
            {
                "_fn": "xpath",
                "_args": [".//option"]
            },
            {
                "_fn": "length"
            }
        ]
    }
}
{
    "number_of_variants": [
        3,
        3
    ]
}

select_nth

Sample HTML

<body>
    <div class="product" id="socks">
        <div class="title">Socks</div>
        <div class="price">123.12</div>
        <div class="description">
            <ul>
                <li class="description-item">Very</li>
                <li class="description-item">Nice</li>
                <li class="description-item">Socks</li>
            </ul>
        </div>
    </div>
</body>

Select the first description item from the array

{
    "price_and_title": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": ["//li[@class='description-item']/text()"]
            },
            {
                "_fn": "select_nth",
                "_args": 0
            }
        ]
    }
}
{
    "price_and_title": "Very"
}

Select the last description item from the array

{
    "price_and_title": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": ["//li[@class='description-item']/text()"]
            },
            {
                "_fn": "select_nth",
                "_args": -1
            }
        ]
    }
}
{
    "price_and_title": "Socks"
}

Math functions

average

Sample HTML

<body>
    <div class="product">
        <div class="price">123</div>
        <div class="price">124</div>
        <div class="price">456</div>
        <div class="price">421</div>
        <div class="price">100</div>
    </div>
</body>

Find the average of all listed prices

{
    "price_average": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": [".//div[@class='price']"]
            },
            {
                "_fn": "xpath_one",
                "_args": ["number(text())"]
            },
            {
                "_fn": "average"
            }
        ]
    }
}
{
    "price_average": 244.8
}

max

Sample HTML

<body>
    <div class="product">
        <div class="price">123</div>
        <div class="price">124</div>
        <div class="price">456</div>
        <div class="price">421</div>
        <div class="price">100</div>
    </div>
</body>

Find the max of all listed prices

{
    "price_max": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": [".//div[@class='price']"]
            },
            {
                "_fn": "xpath_one",
                "_args": ["number(text())"]
            },
            {
                "_fn": "max"
            }
        ]
    }
}
{
    "price_max": 456.0
}

min

Sample HTML

<body>
    <div class="product">
        <div class="price">123</div>
        <div class="price">124</div>
        <div class="price">456</div>
        <div class="price">421</div>
        <div class="price">100</div>
    </div>
</body>

Find the average of all listed prices

{
    "price_min": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": [".//div[@class='price']"]
            },
            {
                "_fn": "xpath_one",
                "_args": ["number(text())"]
            },
            {
                "_fn": "min"
            }
        ]
    }
}
{
    "price_min": 100.0
}

product

Sample HTML

<body>
    <div class="product">
        <property class="colors">
            <option class="color">Red</option>
            <option class="color">Green</option>
            <option class="color">Blue</option>
        </property>
        <property class="sizes">
            <option class="size">S</option>
            <option class="size">M</option>
            <option class="size">L</option>
            <option class="size">XL</option>
        </property>
    </div>
</body>

Get the count of different product variants

{
    "number_of_variants": {
        "_fns": [
            {
                "_fn": "xpath",
                "_args": [".//property"]
            },
            {
                "_fn": "xpath",
                "_args": [".//option"]
            },
            {
                "_fn": "length"
            },
            {
                "_fn": "product"
            }
        ]
    }
}
{
    "number_of_variants": 12
}
PreviousList of functionsNextDownload Images

Last updated 10 months ago

Was this helpful?