Start using Web Scraper API
Last updated
Was this helpful?
Last updated
Was this helpful?
Begin by creating an account on the . After that, select a pricing plan or pick a free trial and create an API user. Your credentials will be your key for user authorization later on.
After creating a user, you'll see a few code examples to test your Web Scraper API. Copy-paste the code into your terminal or preferred tool. For manual testing, we recommend using Postman. You can start scraping Amazon, Google, or any other target right away. Here are the cURL examples for your first request:
Amazon:
Google:
Other:
Replace the USERNAME and PASSWORD with the API credentials you’ve just created, and then , Postman, or any other setup.
See this short video tutorial demonstrating how to use Web Scraper API:
There are three different methods you can use to integrate Web Scraper API:
If you want to extend the API functionality in your projects, you can use the following features integrated into Web Scraper API:
Test our Web Scraper API in the Playground, which is accessible via the .
– provides a synchronous method where you have to keep the connection open until the job is finished.
– enables an asynchronous method where you’ll have to make another request to the API to retrieve results once the job is finished.
– offers a synchronous method where you can use our endpoint as a proxy.
For more information, please refer to the respective integration pages on our or this comprehensive .
Web Scraper API can effectively gather public data from any website, including e-commerce, search engines, online travel agencies, real estate platforms, and others. Visit to learn more about the parameters you can use and see code examples.
For using features such as JavaScript rendering and user agents, explore the pages in our documentation.
– generates ready-to-use code for web scraping in seconds. This feature can be accessed via API Playground on our dashboard. We also have a collection of pre-made prompts and code samples to help you get public data from different targets even faster. It’s always available for you in the .
– renders JavaScript and lets you define custom browser instructions, such as entering text, clicking elements, scrolling pages, and more.
– enables you to create your own parsing and data processing logic that’s executed on a raw scraping result.
– crawls any site, allowing you to select useful content and receive it in bulk. You can use it to perform URL discovery, crawl all pages on a site, index all URLs on a domain, and more.
– provides a way for you to automate recurring scraping and parsing tasks by creating schedules.
– allows you to automate browser actions – like clicking, scrolling, filling out forms, or waiting for elements – before extracting data from a webpage. The feature is accessible in the API playground.