Scraper APIs support three integration methods, each one with its unique benefits:
- Realtime. This method is synchronous. This means that upon sending your job submission request, you will have to keep the connection open until we successfully finish your job or return an error. This integration method is great if you would like to send JSON payloads with scraping and parsing job descriptions, including some advanced scraping parameters, to our API.
- Proxy Endpoint. This method is also synchronous (like Realtime), but instead of using our service via a RESTful interface, you can use our endpoint like a proxy. Use Proxy Endpoint if you've used proxies before and would just like to get unblocked content from us.
- Push-Pull. This integration method is asynchronous. This means that upon submitting your job, we will quickly return you a JSON with your job info, including the URLs for downloading the result/checking the job status. Once we are done processing your job, we will
POSTa notification about job completion to your server, at which point you can go ahead and download the results. This integration method also enables uploading results straight to your cloud storage (AWS S3 or Google Cloud Storage).
The TTL of all API connections is equal to 150 seconds.