Web Scraper API integration methods
Last updated
Was this helpful?
Last updated
Was this helpful?
Web Scraper API integration methods are the different ways you can use our Web Scraper API within your infrastructure and make calls to the API. There are three methods, each having its own benefit. See this blog post for an in-depth look.
This method is easier to implement and use, yet it consumes more infrastructure resources.
Realtime – you have to keep the connection open until the job is finished. This technique is great for sending JSON payloads with scraping and parsing descriptions, including advanced scraping parameters.
Proxy Endpoint – allows you to use our endpoint like a proxy server. You can use this technique if you’re more familiar with proxies and would like to just get unblocked content.
This method is best for large-scale web scraping as it offers more functionality and doesn’t consume additional infrastructure resources.
Push-Pull – you’ll have to send another request to our API to retrieve job results. This approach also enables you to retrieve scraped results straight to your cloud storage, such as AWS S3 or Google Cloud Storage.