Cloud Integration
Scraper API job results are stored in our storage. You can get your results from our storage by
GETting the /results endpoint.As an alternative, we can upload the results onto your cloud storage. This way, you don't have to make extra requests to fetch results - everything goes directly to your storage bucket.
Currently, we support Amazon S3 and Google Cloud Storage. If you would like to use a different type of storage, please contact your account manager to discuss the feature delivery timeline.
The upload path looks like this:
YOUR_BUCKET_NAME/job_ID.json. You will find the job ID in the response that you receive from us after submitting a job. In this example, the job ID is 6942377241502298113.Parameter | Description | Valid values |
|---|---|---|
storage_type | Your cloud storage type. | s3 (AWS S3);
gcs (Google Cloud Storage). |
storage_url | Your cloud storage name. | Any s3 or gcs bucket name. |
To get your job results uploaded to your Amazon S3 bucket, please set up access permissions for our service. To do that, go to
https://s3.console.aws.amazon.com/ > S3 > Storage > Bucket Name (if don't have one, create a new one) > Permissions > Bucket Policy
You can find the bucket policy attached below or in the code sample area.
s3_bucket_policy.json
774B
Code
s3 bucket policy
Don't forget to change the bucket name under
YOUR_BUCKET_NAME. This policy allows us to write to your bucket, give access to uploaded files to you, and know the location of the bucket.{
"Version": "2012-10-17",
"Id": "Policy1577442634787",
"Statement": [
{
"Sid": "Stmt1577442633719",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
},
"Action": "s3:GetBucketLocation",
"Resource": "arn:aws:s3:::YOUR_BUCKET_NAME"
},
{
"Sid": "Stmt1577442633719",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::324311890426:user/oxylabs.s3.uploader"
},
"Action": [
"s3:PutObject",
"s3:PutObjectAcl"
],
"Resource": "arn:aws:s3:::YOUR_BUCKET_NAME/*"
}
]
}
To get your job results uploaded to your Google Cloud Storage bucket, please set up special permissions for our service. To do that, create a custom role with the
storage.objects.create permission, and assign it to the Oxylabs service account email [email protected]. 
Last modified 3mo ago