Web Scraper API features

Our Web Scraper API comes with freely available features that you can use to scale, speed up, and improve your public data-gathering efforts. Refer to the following list of features and visit their documentation pages for in-depth configuration steps.

OxyCopilot

Develop web scrapers and parsers with an AI-powered feature, OxyCopilot, via Scraper API Playground by simply providing target URLs and writing your needs in plain English. To learn more about how OxyCopilot works and explore ready-to-use prompts, visit the OxyCopilot prompts and code samples library available on our website.

Cloud integration

The cloud integration feature enables you to automatically retrieve job results directly to your Amazon S3, Google Cloud Storage, Alibaba OSS, or other S3-compatible storage. This way, you don’t have to make additional requests to get the data from us.

Batch queries

For efficient scraping operations, Web Scraper API allows you to submit up to 5,000 query or URL parameters per batch. Head to our documentation to learn more.

Headless Browser

With the Headless Browser feature, you can render JavaScript on web pages, manipulate DOM, and execute browser actions like entering text, clicking elements, scrolling, and more.

Custom Parser

When you want to parse the HTML of a web page, you can do so with Custom Parser by crafting your own parsing and data processing logic. This feature is especially valuable in scenarios where you want to retrieve parsed data, but we don’t have a dedicated parser for a specific target website.

Scheduler

For automatic execution of recurring scraping and parsing jobs, you can leverage the Scheduler feature to create schedules. We recommend using this feature together with cloud integration to retrieve data at specified intervals.

Browser instructions

Instead of coding browser instructions manually, you can either use our intuitive step-by-step interface in the Playground or generate them with AI using a simple natural language prompt. The system then automatically generates the necessary code, which you can download as a structured JSON file for seamless integration into your API requests.

XHR request capturing

Sometimes it is more convenient to extract the required data from one or more of the Fetch/XHR requests that a browser makes while loading the web page, rather than parsing the HTML. Fetch/XHR request capturing is a feature that lets you retrieve these requests as structured JSON data from dynamic content sources. ​

---

Head back to the dashboard