Web Scraper API integration methods

Learn about the different integration methods available for the Web Scraper API.

Web Scraper API integration methods are the different ways you can use our Web Scraper API within your infrastructure and make calls to the API. There are three methods, each having its own benefit. See this blog post for an in-depth look.

Synchronous integration

This method is easier to implement and use, yet it consumes more infrastructure resources.

• Realtime – you have to keep the connection open until the job is finished. This technique is great for sending JSON payloads with scraping and parsing descriptions, including advanced scraping parameters.

• Proxy Endpoint – allows you to use our endpoint like a proxy server. You can use this technique if you’re more familiar with proxies and would like to just get unblocked content.

Asynchronous integration

This method is best for large-scale web scraping as it offers more functionality and doesn’t consume additional infrastructure resources.

Push-Pull – you’ll have to send another request to our API to retrieve job results. This approach also enables you to retrieve scraped results straight to your cloud storage, such as AWS S3, Google Cloud Storage, Alibaba OSS, and other S3-compatible storage.

For more information about each integration method, please refer to the respective pages of our documentation.

---

Head back to the dashboard