Generic Target
Last updated
Was this helpful?
Last updated
Was this helpful?
Scrape other websites (Home Depot, Idealo, Zillow, , , etc.) with our universal
source. It accepts URLs along with .
In this example, the API will retrieve an e-commerce product page.
We use synchronous integration method in our examples. If you would like to use or asynchronous integration, refer to the section.
source
Sets the scraper.
universal
url
Direct URL (link) to any page.
-
callback_url
-
- mandatory parameter
geo_location
-
render
-
browser_instructions
-
parse
Returns parsed data when set to true
, as long as a dedicated parser exists for the submitted URL's page type.
false
parsing_instructions
-
context
:
headers
-
context
:
cookies
-
context
:
session_id
If you want to use the same proxy with multiple requests, you can do so by using this parameter. Just set your session to any string you like, and we will assign a proxy to this ID, and keep it for up to 10 minutes. After that, if you make another request with the same session ID, a new proxy will be assigned to that particular session ID.
-
context
:
http_method
get
user_agent_type
desktop
context
:
content
Base64-encoded POST
request body. It is only useful if http_method
is set to post
.
-
content_encoding
base64
context
:
follow_redirects
Set to true
to enable scraper to follow redirects. By default, redirects are followed up to a limit of 10 links, treating the entire chain as one scraping job.
true
context
:
successful_status_codes
Define a custom HTTP response code (or a few of them), upon which we should consider the scrape successful and return the content to you. May be useful if you want us to return the 503 error page or in some other non-standard cases.
-
All parameters
In this example, all available parameters are included (though not always necessary or compatible within the same request), to give you an idea on how to format your requests.
URL to your callback endpoint. .
These are the parameters of our .
Sets the proxy's geo location to retrieve data. Find supported locations .
Enables JavaScript rendering when set to html
. . NOTE: If you are observing low success rates or retrieve empty content, please try adding this parameter.
Define your own browser instructions that are executed when rendering JavaScript. .
Define your own parsing and data transformation logic that will be executed on an HTML scraping result. Read more: .
Pass your own headers. Learn more.
Pass your own cookies. Learn more .
Set it to post
if you would like to make a POST
request to your target URL via E-commerce Scraper API. Learn more .
Device type and browser. The full list can be found
Add this parameter if you are downloading images. Learn more .