> For the complete documentation index, see [llms.txt](https://developers.oxylabs.io/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://developers.oxylabs.io/products/pt-br/web-scraper-api/integration-methods/push-pull.md). # Push-Pull Push-Pull é nosso método de integração recomendado para lidar de forma confiável com grandes volumes de dados. {% hint style="success" %} Visite o repositório GitHub da Oxylabs para um exemplo completo e funcional de [**integração Push-Pull em Python**](https://github.com/oxylabs/product-integrations/tree/master/scraper-apis/Python/push_pull). {% endhint %} Push-Pull é um método de integração assíncrono. Ao enviar o job, você receberá prontamente um `JSON` resposta contendo todos os detalhes do job, incluindo parâmetros do job, ID e URLs para download dos resultados e verificação de status. Assim que seu job for processado, enviaremos uma atualização via um `JSON` payload enviado para seu servidor, se você forneceu um [**callback**](#callback) URL. Os resultados permanecem disponíveis para recuperação por **pelo menos 24 horas** após a conclusão. Com Push-Pull, você pode enviar seus resultados diretamente para seu [**armazenamento em nuvem**](/products/pt-br/web-scraper-api/features/result-processing-and-storage/cloud-storage.md) (Google Cloud Storage, AWS S3, Alibaba Cloud OSS, ou outro armazenamento compatível com S3). {% hint style="info" %} Se você preferir não configurar um serviço para notificações de callback recebidas, você pode simplesmente recuperar seus resultados periodicamente (*polling*). {% endhint %} Você também pode explorar como Push-Pull funciona usando [**Postman**](/integrations/pt-br/integracoes-da-web-scraper-api/postman.md). ## Job único ### Endpoint Este endpoint aceita apenas um único `query` ou `URL` valor. ``` POST https://data.oxylabs.io/v1/queries ``` ### Entrada Forneça os parâmetros do job em um payload JSON, como mostrado nos exemplos abaixo. Os exemplos em Python e PHP incluem comentários para clareza. {% tabs %} {% tab title="cURL" %} ```shell curl --user "user:pass1" \\ 'https://data.oxylabs.io/v1/queries' \\ -H "Content-Type: application/json" \\ -d '{"source": "ENTER_SOURCE_HERE", "url": "https://www.example.com", "geo_location": "United States", "callback_url": "https://your.callback.url", "storage_type": "s3", "storage_url": "s3://your.storage.bucket.url"}' ``` {% endtab %} {% tab title="Python" %} ```python import requests from pprint import pprint # Structure payload. payload = { "source": "ENTER_SOURCE_HERE", # Source you choose e.g. "universal" "url": "https://www.example.com", # Check speficic source if you should use "url" or "query" "geo_location": "United States", # Some sources accept zip-code or cooprdinates #"render" : "html", # Uncomment you want to render JavaScript within the page #"render" : "png", # Uncomment if you want to take a screenshot of a scraped web page #"parse" : true, # Check what sources support parsed data #"callback_url": "https://your.callback.url", #required if using callback listener "callback_url": "https://your.callback.url", "storage_type": "s3", "storage_url": "s3://your.storage.bucket.url" } # Get response. response = requests.request( 'POST', 'https://data.oxylabs.io/v1/queries', auth=('YOUR_USERNAME', 'YOUR_PASSWORD'), #Your credentials go here json=payload, ) # Print prettified response to stdout. pprint(response.json()) ``` {% endtab %} {% tab title="PHP" %} ```php 'ENTER_SOURCE_HERE', //Source you choose e.g. "universal" 'url' => 'https://www.example.com', // Check speficic source if you should use "url" or "query" 'geo_location' => 'United States', //Some sources accept zip-code or cooprdinates //'render' : 'html', // Uncomment you want to render JavaScript within the page //'render' : 'png', // Uncomment if you want to take a screenshot of a scraped web page //'parse' : TRUE, // Check what sources support parsed data //'callback_url' => 'https://your.callback.url', //required if using callback listener 'callback_url': 'https://your.callback.url', 'storage_type' => 's3', 'storage_url' => 's3://your.storage.bucket.url' ); $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, "https://data.oxylabs.io/v1/queries"); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params)); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_USERPWD, "YOUR_USERNAME" . ":" . "YOUR_PASSWORD"); //Your credentials go here $headers = array(); $headers[] = "Content-Type: application/json"; curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); $result = curl_exec($ch); echo $result; if (curl_errno($ch)) { echo 'Error:' . curl_error($ch); } curl_close ($ch); ?> ``` {% endtab %} {% tab title="C#" %} ```csharp using System; using System.Collections.Generic; using System.Net.Http; using System.Net.Http.Json; using System.Threading.Tasks; namespace OxyApi { class Program { static async Task Main() { const string Username = "YOUR_USERNAME"; const string Password = "YOUR_PASSWORD"; var parameters = new Dictionary() { { "source", "ENTER_SOURCE_HERE" }, { "url", "https://example.com" }, { "geo_location", "United States" }, { "callback_url", "https://your.callback.url" }, }; var client = new HttpClient(); Uri baseUri = new Uri("https://data.oxylabs.io"); client.BaseAddress = baseUri; var requestMessage = new HttpRequestMessage(HttpMethod.Post, "/v1/queries"); requestMessage.Content = JsonContent.Create(parameters); var authenticationString = $"{Username}:{Password}"; var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.ASCIIEncoding.UTF8.GetBytes(authenticationString)); requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString); var response = await client.SendAsync(requestMessage); var contents = await response.Content.ReadAsStringAsync(); Console.WriteLine(contents); } } } ``` {% endtab %} {% tab title="Golang" %} ```go package main import ( "bytes" "encoding/json" "fmt" "io/ioutil" "net/http" ) func main() { const Username = "YOUR_USERNAME" const Password = "YOUR_PASSWORD" payload := map[string]string{ "source": "ENTER_SOURCE_HERE", "url": "https://example.com", "geo_location": "United States", "callback_url": "https://your.callback.url", } jsonValue, _ := json.Marshal(payload) client := &http.Client{} request, _ := http.NewRequest("POST", "https://data.oxylabs.io/v1/queries", bytes.NewBuffer(jsonValue), ) request.Header.Add("Content-type", "application/json") request.SetBasicAuth(Username, Password) response, _ := client.Do(request) responseText, _ := ioutil.ReadAll(response.Body) fmt.Println(string(responseText)) } ``` {% endtab %} {% tab title="Java" %} ```java package org.example; import okhttp3.*; import org.json.JSONObject; public class Main implements Runnable { private static final String AUTHORIZATION_HEADER = "Authorization"; public static final String USERNAME = "YOUR_USERNAME"; public static final String PASSWORD = "YOUR_PASSWORD"; public void run() { JSONObject jsonObject = new JSONObject(); jsonObject.put("source", "ENTER_SOURCE_HERE"); jsonObject.put("url", "https://example.com"); jsonObject.put("geo_location", "United States"); jsonObject.put("callback_url", "https://your.callback.url"); Authenticator authenticator = (route, response) -> { String credential = Credentials.basic(USERNAME, PASSWORD); return response .request() .newBuilder() .header(AUTHORIZATION_HEADER, credential) .build(); }; var client = new OkHttpClient.Builder() .authenticator(authenticator) .build(); var mediaType = MediaType.parse("application/json; charset=utf-8"); var body = RequestBody.create(jsonObject.toString(), mediaType); var request = new Request.Builder() .url("https://data.oxylabs.io/v1/queries") .post(body) .build(); try (var response = client.newCall(request).execute()) { assert response.body() != null; System.out.println(response.body().string()); } catch (Exception exception) { System.out.println("Error: " + exception.getMessage()); } System.exit(0); } public static void main(String[] args) { new Thread(new Main()).start(); } } ``` {% endtab %} {% tab title="Node.js" %} ```shell import fetch from 'node-fetch'; const username = 'YOUR_USERNAME'; const password = 'YOUR_PASSWORD'; const body = { source: 'ENTER_SOURCE_HERE', url: 'https://www.example.com', geo_location: 'United States', callback_url: 'https://your.callback.url', }; const response = await fetch('https://data.oxylabs.io/v1/queries', { method: 'post', body: JSON.stringify(body), headers: { 'Content-Type': 'application/json', 'Authorization': 'Basic ' + Buffer.from(`${username}:${password}`).toString('base64'), } }); console.log(await response.json()); ``` {% endtab %} {% endtabs %} ### Saída A API responderá com um JSON contendo as informações do job, semelhante a este: ```json { "callback_url": "https://your.callback.url", "client_id": 5, "context": [ { "key": "results_language", "value": null }, { "key": "safe_search", "value": null }, { "key": "tbm", "value": null }, { "key": "cr", "value": null }, { "key": "filter", "value": null } ], "created_at": "2024-06-26 00:00:01", "geo_location": "United States", "id": "12345678900987654321", "limit": 10, "locale": null, "pages": 1, "parse": false, "render": null, "url": "https://www.example.com", "source": "universal", "start_page": 1, "status": "pending", "storage_type": "s3", "storage_url": "YOUR_BUCKET_NAME/12345678900987654321.json", "subdomain": "www", "updated_at": "2024-06-26 00:00:01", "user_agent_type": "desktop", "_links": [ { "rel": "self", "href": "http://data.oxylabs.io/v1/queries/12345678900987654321", "method": "GET" }, { "rel": "results", "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results", "method": "GET" } ] } ``` ### Dicionário de dados Para descrições detalhadas dos parâmetros de entrada do job, consulte a tabela abaixo ou a documentação específica dos scrapers de seu interesse. | Chave | Descrição | Tipo | | ------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | -------------------- | | `created_at` | A data e hora em que o job foi criado. | Cadeia de caracteres | | `client_id` | O ID numérico associado ao nome de usuário do cliente que está fazendo a requisição. | Cadeia de caracteres | | `client_notes` | Notas enviadas pelo cliente ao enviar um job. | Cadeia de caracteres | | `content_encoding` | Adicione este parâmetro se você estiver baixando imagens. Saiba mais [**aqui**](/products/pt-br/web-scraper-api/features/result-processing-and-storage/output-types/download-images.md)**.** | Cadeia de caracteres | | `id` | O ID único do job. | Cadeia de caracteres | | `statuses` | O código de status do job de scraping ou parsing. Você pode ver os códigos de status descritos [**aqui**](/products/pt-br/web-scraper-api/response-codes.md). | Inteiro | | `status` | O status do job. `pending` significa que o job ainda está sendo processado. `done` significa que concluímos o job. `faulted` significa que encontramos erros ao tentar concluir o job -- e desistimos dele. | Cadeia de caracteres | | `subdomain` | O subdomínio do site. | Cadeia de caracteres | | `updated_at` | A data e hora da última atualização do job. Para jobs concluídos (`status` é `done` ou `faulted`), esta data e hora indica quando o job foi concluído. | Cadeia de caracteres | | `links` | A lista de links relacionada à entrada fornecida. | Array JSON | | `links`:`rel` | O tipo do link. `self` A URL contém os metadados do job, enquanto `results` A URL contém os resultados do job. | Cadeia de caracteres | | `links`:`href` | A URL do recurso. | Cadeia de caracteres | | `links`:`method` | O método HTTP que deve ser usado para interagir com uma URL específica. | Cadeia de caracteres | ## Callback O callback é uma `POST` requisição que enviamos para sua máquina, informando que a tarefa de extração de dados foi concluída e fornecendo uma URL para baixar o conteúdo extraído. Isso significa que você não precisa [**verificar o status do job**](#check-job-status) manualmente. Assim que os dados estiverem aqui, avisaremos você, e tudo o que você precisa fazer agora é [**recuperá-los**](#retrieve-job-content). ### Entrada {% tabs %} {% tab title="Python" %} ```python # This is a simple Sanic web server with a route listening for callbacks on localhost:8080. # It will print job results to stdout. import requests from pprint import pprint from sanic import Sanic, response AUTH_TUPLE = ('user', 'pass1') app = Sanic() # Define /job_listener endpoint that accepts POST requests. @app.route('/job_listener', methods=['POST']) async def job_listener(request): try: res = request.json links = res.get('_links', []) for link in links: if link['rel'] == 'results': # Sanic is async, but requests are synchronous, to fully take # advantage of Sanic, use aiohttp. res_response = requests.request( method='GET', url=link['href'], auth=AUTH_TUPLE, ) pprint(res_response.json()) break except Exception as e: print("Listener exception: {}".format(e)) return response.json(status=200, body={'status': 'ok'}) if __name__ == '__main__': app.run(host='0.0.0.0', port=8080) ``` {% endtab %} {% tab title="PHP" %} ```php ``` {% endtab %} {% tab title="C#" %} ```csharp using Microsoft.AspNetCore.Builder; using Microsoft.AspNetCore.Hosting; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Hosting; using System; using System.Collections.Generic; using System.Net.Http; namespace OxyApiWeb { public class Callback { public Link[] _links { get; set; } } public class Link { public string rel { get; set; } public string href { get; set; } } public class Startup { private const string USERNAME = "YOUR_USERNAME"; private const string PASSWORD = "YOUR_PASSWORD"; public Startup(IConfiguration configuration) { Configuration = configuration; client = new HttpClient(); } public IConfiguration Configuration { get; } private HttpClient client; public void ConfigureServices(IServiceCollection services) { services.AddControllers(); } public void Configure(IApplicationBuilder app, IWebHostEnvironment env) { if (env.IsDevelopment()) { app.UseDeveloperExceptionPage(); } app.UseRouting(); app.UseAuthorization(); app.UseEndpoints(endpoints => { endpoints.MapPost("/job_listener", async context => { var callback = await System.Text.Json.JsonSerializer.DeserializeAsync(context.Request.Body); foreach (var link in callback._links) { if (link.rel != "results") { continue; } var requestMessage = new HttpRequestMessage(HttpMethod.Get, new Uri(link.href)); var authenticationString = $"{USERNAME}:{PASSWORD}"; var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.ASCIIEncoding.UTF8.GetBytes(authenticationString)); requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString); var response = await client.SendAsync(requestMessage); var contents = await response.Content.ReadAsStringAsync(); Console.WriteLine(contents); } var okMessage = new Dictionary() { { "message", "ok" } }; await System.Text.Json.JsonSerializer.SerializeAsync(context.Response.Body, okMessage); }); }); } } } ``` {% endtab %} {% tab title="Golang" %} ```go package main import ( "fmt" "github.com/labstack/echo/v4" "io/ioutil" "net/http" ) const Username = "YOUR_USERNAME" const Password = "YOUR_PASSWORD" type Callback struct { Links []Link `json:"_links"` } type Link struct { Href string `json:"href"` Method string `json:"method"` Rel string `json:"rel"` } func main() { echoServer := echo.New() client := &http.Client{} echoServer.POST("/job_listener", func(context echo.Context) error { callback := new(Callback) if err := context.Bind(callback); err != nil { return err } for _, link := range callback.Links { if link.Rel != "results" { continue } request, _ := http.NewRequest("GET", link.Href, nil, ) request.Header.Add("Content-type", "application/json") request.SetBasicAuth(Username, Password) response, _ := client.Do(request) responseText, _ := ioutil.ReadAll(response.Body) fmt.Println(string(responseText)) } return context.JSON(http.StatusOK, map[string]string { "status": "ok" }) }) echoServer.Logger.Fatal(echoServer.Start(":8080")) } ``` {% endtab %} {% tab title="Java" %} ```java package org.example; import okhttp3.*; import com.sun.net.httpserver.HttpServer; import org.apache.commons.io.IOUtils; import org.json.JSONArray; import org.json.JSONObject; import java.io.IOException; import java.io.OutputStream; import java.net.InetSocketAddress; import java.nio.charset.StandardCharsets; import java.util.Map; import java.util.Objects; public class Main implements Runnable { private static final String AUTHORIZATION_HEADER = "Authorization"; public static final String USERNAME = "YOUR_USERNAME"; public static final String PASSWORD = "YOUR_PASSWORD"; public void run() { HttpServer server = null; try { server = HttpServer.create(new InetSocketAddress("0.0.0.0", 8080), 0); } catch (IOException exception) { exception.printStackTrace(); System.exit(1); } Authenticator authenticator = (route, response) -> { String credential = Credentials.basic(USERNAME, PASSWORD); return response .request() .newBuilder() .header(AUTHORIZATION_HEADER, credential) .build(); }; var client = new OkHttpClient.Builder() .authenticator(authenticator) .build(); server.createContext("/job_listener", exchange -> { var requestBody = IOUtils.toString(exchange.getRequestBody(), StandardCharsets.UTF_8); JSONObject requestJson = new JSONObject(requestBody); JSONArray links = requestJson.getJSONArray("_links"); for (var link : links.toList()) { var linkMap = (Map)link; if (!Objects.equals(linkMap.get("rel"), "results")) { continue; } var request = new Request.Builder() .url((String) linkMap.get("href")) .get() .build(); try (var response = client.newCall(request).execute()) { assert response.body() != null; System.out.println(response.body().string()); } catch (Exception exception) { System.out.println("Error: " + exception.getMessage()); } } var responseJson = new JSONObject(); responseJson.put("status", "ok"); exchange.sendResponseHeaders(200, responseJson.toString().length()); OutputStream responseBody = exchange.getResponseBody(); responseBody.write(responseJson.toString().getBytes()); responseBody.flush(); responseBody.close(); exchange.close(); }); server.setExecutor(null); server.start(); } public static void main(String[] args) { new Thread(new Main()).start(); } } ``` {% endtab %} {% tab title="Node.js" %} ```shell import express from 'express' import fetch from 'node-fetch'; const username = 'YOUR_USERNAME'; const password = 'YOUR_PASSWORD'; const app = express(); app.use(express.json()); app.post('/job_listener', async(request, response) => { for (const index in request.body._links) { const link = request.body._links[index]; if (link.rel !== 'results') { continue; } const jobResultResponse = await fetch(link.href, { method: 'get', headers: { 'Content-Type': 'application/json', 'Authorization': 'Basic ' + Buffer.from(`${username}:${password}`).toString('base64'), } }); console.log(await jobResultResponse.json()); } response.send({status: 'ok'}); }); app.listen(8080); ``` {% endtab %} {% endtabs %} ### Saída ```json { "created_at":"2019-10-01 00:00:01", "updated_at":"2019-10-01 00:00:15", "locale":null, "client_id":163, "user_agent_type":"desktop", "source":"google_shopping_search", "pages":1, "subdomain":"www", "status":"done", "start_page":1, "parse":0, "render":null, "priority":0, "ttl":0, "origin":"api", "persist":true, "id":"12345678900987654321", "callback_url":"http://your.callback.url/", "query":"adidas", "limit":10, "geo_location":null, {...} "_links":[ { "href":"https://data.oxylabs.io/v1/queries/12345678900987654321", "method":"GET", "rel":"self" }, { "href":"https://data.oxylabs.io/v1/queries/12345678900987654321/results", "method":"GET", "rel":"results" } ], } ``` ## Verificar status do job Se você forneceu uma URL de callback válida ao enviar seu job, nós o notificaremos quando ele for concluído enviando um `JSON` payload para a URL de callback especificada. Esse payload indicará que o job foi concluído e seu status definido como `done`. No entanto, se você enviou um job sem usar [**serviço de callback**](#callback), você pode verificar o status do job manualmente. Recupere a URL do `href` campo na `rel:self` seção da mensagem de resposta recebida após o envio do job. A URL para verificar o status do job será semelhante à seguinte: `http://data.oxylabs.io/v1/queries/12345678900987654321`. Consultar esta URL retornará as informações do job, incluindo seu status atual, `status`. ### Endpoint ``` GET https://data.oxylabs.io/v1/queries/{id} ``` ### Entrada {% tabs %} {% tab title="cURL" %} ```bash curl --user "user:pass1" \\ 'http://data.oxylabs.io/v1/queries/12345678900987654321' ``` {% endtab %} {% tab title="Python" %} ```python import requests from pprint import pprint # Get response from stats endpoint. response = requests.request( method='GET', url='http://data.oxylabs.io/v1/queries/12345678900987654321', auth=('user', 'pass1'), ) # Print prettified JSON response to stdout. pprint(response.json()) ``` {% endtab %} {% tab title="PHP" %} ```php ``` {% endtab %} {% tab title="C#" %} ```csharp using System; using System.Collections.Generic; using System.Net.Http; using System.Net.Http.Json; using System.Threading.Tasks; namespace OxyApi { class Program { static async Task Main() { const string JobId = "12345678900987654321"; const string Username = "YOUR_USERNAME"; const string Password = "YOUR_PASSWORD"; var client = new HttpClient(); Uri baseUri = new Uri("https://data.oxylabs.io"); client.BaseAddress = baseUri; var requestMessage = new HttpRequestMessage(HttpMethod.Get, $"/v1/queries/{JobId}"); var authenticationString = $"{Username}:{Password}"; var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.ASCIIEncoding.UTF8.GetBytes(authenticationString)); requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString); var response = await client.SendAsync(requestMessage); var contents = await response.Content.ReadAsStringAsync(); Console.WriteLine(contents); } } } ``` {% endtab %} {% tab title="Golang" %} ```go package main import ( "fmt" "io/ioutil" "net/http" ) func main() { const JobId = "12345678900987654321" const Username = "YOUR_USERNAME" const Password = "YOUR_PASSWORD" client := &http.Client{} request, _ := http.NewRequest("GET", fmt.Sprintf("https://data.oxylabs.io/v1/queries/%s", JobId), nil, ) request.Header.Add("Content-type", "application/json") request.SetBasicAuth(Username, Password) response, _ := client.Do(request) responseText, _ := ioutil.ReadAll(response.Body) fmt.Println(string(responseText)) } ``` {% endtab %} {% tab title="Java" %} ```java package org.example; import okhttp3.*; public class Main implements Runnable { private static final String AUTHORIZATION_HEADER = "Authorization"; private static final String JOB_ID = "12345678900987654321"; public static final String USERNAME = "YOUR_USERNAME"; public static final String PASSWORD = "YOUR_PASSWORD"; public void run() { Authenticator authenticator = (route, response) -> { String credential = Credentials.basic(USERNAME, PASSWORD); return response .request() .newBuilder() .header(AUTHORIZATION_HEADER, credential) .build(); }; var client = new OkHttpClient.Builder() .authenticator(authenticator) .build(); var request = new Request.Builder() .url(String.format("https://data.oxylabs.io/v1/queries/%s", JOB_ID)) .get() .build(); try (var response = client.newCall(request).execute()) { assert response.body() != null; System.out.println(response.body().string()); } catch (Exception exception) { System.out.println("Error: " + exception.getMessage()); } System.exit(0); } public static void main(String[] args) { new Thread(new Main()).start(); } } ``` {% endtab %} {% tab title="Node.js" %} ```shell import fetch from 'node-fetch'; const jobId = '12345678900987654321'; const username = 'YOUR_USERNAME'; const password = 'YOUR_PASSWORD'; const response = await fetch(`https://data.oxylabs.io/v1/queries/${jobId}`, { method: 'get', headers: { 'Content-Type': 'application/json', 'Authorization': 'Basic ' + Buffer.from(`${username}:${password}`).toString('base64'), } }); console.log(await response.json()); ``` {% endtab %} {% endtabs %} ### Saída Após a conclusão do job, a API responderá com as informações da query em formato JSON. O status do job será alterado para `done`, indicando que o job foi concluído. Você pode recuperar o conteúdo consultando um dos links fornecidos. Além disso, a resposta incluirá o timestamp da última atualização do job, permitindo acompanhar seu tempo de processamento. ```json { "client_id": 5, "context": [ { "key": "results_language", "value": null }, { "key": "safe_search", "value": null }, { "key": "tbm", "value": null }, { "key": "cr", "value": null }, { "key": "filter", "value": null } ], "created_at": "2019-10-01 00:00:01", "geo_location": null, "id": "7173957294344910849", "limit": 10, "locale": null, "pages": 1, "parse": false, "render": null, "query": "adidas", "source": "google_shopping_search", "start_page": 1, "status": "done", "subdomain": "www", "updated_at": "2019-10-01 00:00:15", "user_agent_type": "desktop", "_links": [ { "rel": "self", "href": "http://data.oxylabs.io/v1/queries/7173957294344910849", "method": "GET" }, { "rel": "results", "href": "http://data.oxylabs.io/v1/queries/7173957294344910849/results", "method": "GET" }, { "rel": "results-html", "href": "http://data.oxylabs.io/v1/queries/7173957294344910849/results?type=raw", "method": "GET" }, { "rel": "results-parsed", "href": "http://data.oxylabs.io/v1/queries/7173957294344910849/results?type=parsed", "method": "GET" }, { "rel": "results-parsed", "href": "http://data.oxylabs.io/v1/queries/7173957294344910849/results?type=png", "method": "GET" } ] } ``` ## Valores de status
ParâmetroDescrição
pendingO job ainda está sendo processado e não foi concluído.
doneO job foi concluído. Você pode recuperar o resultado consultando a URL fornecida no href campo sob a rel:results seção, por exemplo: http://data.oxylabs.io/v1/queries/12345678900987654321/results.
faultedHouve um problema com o job, e não conseguimos concluí-lo. Você não será cobrado por nenhum faulted job.
## Recuperar conteúdo do job Quando o job estiver pronto para ser recuperado, você pode usar a URL fornecida na resposta sob a `rel:results` seção. A URL ficará assim: `http://data.oxylabs.io/v1/queries/7173957294344910849/results`. ### Endpoints Você pode recuperar diferentes tipos de resultado usando os seguintes endpoints: ``` GET https://data.oxylabs.io/v1/queries/{job_id}/results ``` ``` GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=raw ``` ``` GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=parsed ``` ``` GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=png ``` ``` GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=xhr ``` ``` GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=markdown ``` Você também pode recuperar [vários tipos de resultado](/products/pt-br/web-scraper-api/features/result-processing-and-storage/output-types/multi-format-output.md) em uma única resposta, por exemplo: ``` GET https://data.oxylabs.io/v1/queries/{job_id}/results?type=parsed,raw ``` ### Entrada Abaixo estão exemplos de código demonstrando como usar o `/results` endpoint: {% tabs %} {% tab title="cURL" %} ```shell curl --user "user:pass1" \\ 'http://data.oxylabs.io/v1/queries/12345678900987654321/results' ``` {% endtab %} {% tab title="Python" %} ```python import requests from pprint import pprint # Get response from stats endpoint. response = requests.request( method='GET', url='http://data.oxylabs.io/v1/queries/12345678900987654321/results', auth=('user', 'pass1'), ) # Print prettified JSON response to stdout. pprint(response.json()) ``` {% endtab %} {% tab title="PHP" %} ```php ``` {% endtab %} {% tab title="C#" %} ```csharp using System; using System.Net.Http; using System.Threading.Tasks; namespace OxyApi { class Program { static async Task Main() { const string JobId = "12345678900987654321"; const string Username = "YOUR_USERNAME"; const string Password = "YOUR_PASSWORD"; var client = new HttpClient(); Uri baseUri = new Uri("https://data.oxylabs.io"); client.BaseAddress = baseUri; var requestMessage = new HttpRequestMessage(HttpMethod.Get, $"/v1/queries/{JobId}/results"); var authenticationString = $"{Username}:{Password}"; var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.ASCIIEncoding.UTF8.GetBytes(authenticationString)); requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString); var response = await client.SendAsync(requestMessage); var contents = await response.Content.ReadAsStringAsync(); Console.WriteLine(contents); } } } ``` {% endtab %} {% tab title="Golang" %} ```go package main import ( "fmt" "io/ioutil" "net/http" ) func main() { const JobId = "12345678900987654321" const Username = "YOUR_USERNAME" const Password = "YOUR_PASSWORD" client := &http.Client{} request, _ := http.NewRequest("GET", fmt.Sprintf("https://data.oxylabs.io/v1/queries/%s/results", JobId), nil, ) request.Header.Add("Content-type", "application/json") request.SetBasicAuth(Username, Password) response, _ := client.Do(request) responseText, _ := ioutil.ReadAll(response.Body) fmt.Println(string(responseText)) } ``` {% endtab %} {% tab title="Java" %} ```java package org.example; import okhttp3.*; public class Main implements Runnable { private static final String AUTHORIZATION_HEADER = "Authorization"; private static final String JOB_ID = "12345678900987654321"; public static final String USERNAME = "YOUR_USERNAME"; public static final String PASSWORD = "YOUR_PASSWORD"; public void run() { Authenticator authenticator = (route, response) -> { String credential = Credentials.basic(USERNAME, PASSWORD); return response .request() .newBuilder() .header(AUTHORIZATION_HEADER, credential) .build(); }; var client = new OkHttpClient.Builder() .authenticator(authenticator) .build(); var request = new Request.Builder() .url(String.format("https://data.oxylabs.io/v1/queries/%s/results", JOB_ID)) .get() .build(); try (var response = client.newCall(request).execute()) { assert response.body() != null; System.out.println(response.body().string()); } catch (Exception exception) { System.out.println("Error: " + exception.getMessage()); } System.exit(0); } public static void main(String[] args) { new Thread(new Main()).start(); } } ``` {% endtab %} {% tab title="Node.js" %} ```shell import fetch from 'node-fetch'; const jobId = '12345678900987654321'; const username = 'YOUR_USERNAME'; const password = 'YOUR_PASSWORD'; const response = await fetch(`https://data.oxylabs.io/v1/queries/${jobId}/results`, { method: 'get', headers: { 'Content-Type': 'application/json', 'Authorization': 'Basic ' + Buffer.from(`${username}:${password}`).toString('base64'), } }); console.log(await response.json()); ``` {% endtab %} {% endtabs %} ### Saída Esta tabela explica os tipos de resultado padrão e outros disponíveis com base nos headers incluídos no payload da requisição da API. | Parâmetro de renderização | Parâmetro de parse | Parâmetro de XHR | Saída padrão | Saída disponível | | ------------------------- | ------------------ | ---------------- | ------------ | ----------------- | | - | - | - | html | html | | `html` | - | - | html | html | | `html` | - | `true` | xhr | html, xhr | | `html` | `true` | `true` | parsed | html, xhr, parsed | | `png` | - | - | png | html, png | | - | `true` | - | parsed | html, parsed | | `html` | `true` | - | parsed | html, parsed | | `png` | `true` | - | png | html, parsed, png | Abaixo está uma resposta de exemplo do `/results` endpoint: ```json { "results": [ { "content": " CONTENT ", "created_at": "2019-10-01 00:00:01", "updated_at": "2019-10-01 00:00:15", "page": 1, "url": "https://www.google.com/search?q=adidas&hl=en&gl=US", "job_id": "12345678900987654321", "status_code": 200 } ] } ``` Os resultados podem ser recuperados automaticamente sem verificar periodicamente o status do job ao configurar [**Callback**](#callback) serviço de callback. Para isso, especifique a URL de um servidor capaz de aceitar requisições HTTP(S) recebidas ao enviar um job. Quando nosso sistema concluir o job, ele irá `POST` enviar um payload JSON para a URL fornecida, e o serviço de Callback baixará os resultados conforme descrito no [**exemplo de implementação de Callback**](#callback). ## Consulta em lote As Scraper APIs permitem enviar até 5.000 `query` ou `url` valores de parâmetro em uma única requisição em lote. ### Endpoint ``` POST https://data.oxylabs.io/v1/queries/batch ``` O sistema processará cada `query` ou `url` enviado como um job separado. Se você fornecer uma URL de callback, receberá uma chamada separada para cada palavra-chave. Caso contrário, nossa resposta inicial conterá job `id`s para todas as palavras-chave. Por exemplo, se você enviar 50 palavras-chave, retornaremos 50 job `id`s exclusivos. {% hint style="info" %} **IMPORTANTE:** Com `/batch` endpoint, você só pode enviar listas de `query`ou `url`valores de parâmetro (dependendo do `fonte` que você usar). Todos os outros parâmetros devem ter valores singulares. {% endhint %} ### Entrada Você precisa enviar os parâmetros da query como um payload JSON. Veja como enviar um job em lote: {% tabs %} {% tab title="cURL" %} ```shell curl --user "user:pass1" \\ 'https://data.oxylabs.io/v1/queries/batch' \ -H 'Content-Type: application/json' \ -d '@keywords.json' ``` {% endtab %} {% tab title="Python" %} ```python import requests import json from pprint import pprint # Get payload from file. with open('keywords.json', 'r') as f: payload = json.loads(f.read()) response = requests.request( 'POST', 'https://data.oxylabs.io/v1/queries/batch', auth=('user', 'pass1'), json=payload, ) # Print prettified response. pprint(response.json()) ``` {% endtab %} {% tab title="PHP" %} ```php ``` {% endtab %} {% tab title="C#" %} ```csharp using System; using System.IO; using System.Net.Http; using System.Text; using System.Threading.Tasks; namespace OxyApi { class Program { static async Task Main() { const string Username = "YOUR_USERNAME"; const string Password = "YOUR_PASSWORD"; var content = File.ReadAllText(@"C:\path\to\keywords.json"); var client = new HttpClient(); var requestMessage = new HttpRequestMessage(HttpMethod.Post, new Uri("https://data.oxylabs.io/v1/queries/batch")); requestMessage.Content = new StringContent(content, Encoding.UTF8, "application/json"); var authenticationString = $"{Username}:{Password}"; var base64EncodedAuthenticationString = Convert.ToBase64String(ASCIIEncoding.UTF8.GetBytes(authenticationString)); requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString); var response = await client.SendAsync(requestMessage); var contents = await response.Content.ReadAsStringAsync(); Console.WriteLine(contents); } } } ``` {% endtab %} {% tab title="Golang" %} ```go package main import ( "bytes" "fmt" "io/ioutil" "net/http" "os" ) func main() { const Username = "YOUR_USERNAME" const Password = "YOUR_PASSWORD" content, err := os.ReadFile("keywords.json") if err != nil { panic(err) } client := &http.Client{} request, _ := http.NewRequest("POST", "https://data.oxylabs.io/v1/queries/batch", bytes.NewBuffer(content), ) request.Header.Add("Content-type", "application/json") request.SetBasicAuth(Username, Password) response, _ := client.Do(request) responseText, _ := ioutil.ReadAll(response.Body) fmt.Println(string(responseText)) } ``` {% endtab %} {% tab title="Java" %} ```java import okhttp3.*; import java.io.IOException; import java.nio.file.Files; import java.nio.file.Path; public class Main implements Runnable { private static final String AUTHORIZATION_HEADER = "Authorization"; public static final String USERNAME = "YOUR_USERNAME"; public static final String PASSWORD = "YOUR_PASSWORD"; public void run() { Path filePath = Path.of("/path/to/keywords.json"); String jsonContent = null; try { jsonContent = Files.readString(filePath); } catch (IOException e) { throw new RuntimeException(e); } Authenticator authenticator = (route, response) -> { String credential = Credentials.basic(USERNAME, PASSWORD); return response .request() .newBuilder() .header(AUTHORIZATION_HEADER, credential) .build(); }; var client = new OkHttpClient.Builder() .authenticator(authenticator) .build(); var mediaType = MediaType.parse("application/json; charset=utf-8"); var body = RequestBody.create(jsonContent, mediaType); var request = new Request.Builder() .url("https://data.oxylabs.io/v1/queries/batch") .post(body) .build(); try (var response = client.newCall(request).execute()) { assert response.body() != null; System.out.println(response.body().string()); } catch (Exception exception) { System.out.println("Error: " + exception.getMessage()); } System.exit(0); } public static void main(String[] args) { new Thread(new Main()).start(); } } ``` {% endtab %} {% tab title="Node.js" %} ```shell import fetch from 'node-fetch'; import fs from 'fs' const username = 'YOUR_USERNAME'; const password = 'YOUR_PASSWORD'; const payload = fs.readFileSync('keywords.json').toString(); const response = await fetch('https://data.oxylabs.io/v1/queries/batch', { method: 'post', body: payload, headers: { 'Content-Type': 'application/json', 'Authorization': 'Basic ' + Buffer.from(`${username}:${password}`).toString('base64'), } }); console.log(await response.json()); ``` {% endtab %} {% endtabs %} Você pode notar que o exemplo de código acima não explica como o payload JSON deve ser formatado e aponta para um arquivo JSON pronto. Abaixo está o conteúdo do `keywords.json` arquivo, contendo vários `query` valores de parâmetro: ```json { "query":[ "adidas", "nike", "reebok" ], "source": "google_shopping_search", "callback_url": "https://your.callback.url" } ``` ...e aqui está um `keywords.json` arquivo de entrada em lote, contendo múltiplas URLs: ```json { "url":[ "https://example.com/url1.html", "https://example.com/url2.html", "https://example.com/url3.html" ], "source": "universal", "callback_url": "https://your.callback.url" } ``` ### Saída A API retornará um objeto JSON, contendo as informações do job de cada job criado. A resposta será semelhante a esta: ```json { "queries": [ { "callback_url": "https://your.callback.url", {...} "created_at": "2019-10-01 00:00:01", "id": "12345678900987654321", {...} "query": "adidas", "source": "google_shopping_search", {...} "rel": "results", "href": "http://data.oxylabs.io/v1/queries/12345678900987654321/results", "method": "GET" } ] }, { "callback_url": "https://your.callback.url", {...} "created_at": "2019-10-01 00:00:01", "id": "12345678901234567890", {...} "query": "nike", "source": "google_shopping_search", {...} "rel": "results", "href": "http://data.oxylabs.io/v1/queries/12345678901234567890/results", "method": "GET" } ] }, { "callback_url": "https://your.callback.url", {...} "created_at": "2019-10-01 00:00:01", "id": "01234567899876543210", {...} "query": "reebok", "source": "google_shopping_search", {...} "rel": "results", "href": "http://data.oxylabs.io/v1/queries/01234567899876543210/results", "method": "GET" } ] } ] } ``` ## Obter lista de endereços IP do Notifier Talvez você queira colocar os IPs que enviam mensagens de callback na whitelist ou obter a lista desses IPs para outros fins. Você pode fazer isso `GET`este endpoint: ### Endpoint ``` GET https://data.oxylabs.io/v1/info/callbacker_ips ``` ### Entrada Os exemplos de código abaixo mostram como você pode acessar o `/callbacker_ips` endpoint: {% tabs %} {% tab title="cURL" %} ```bash curl --user "user:pass1" \\ 'https://data.oxylabs.io/v1/info/callbacker_ips' ``` {% endtab %} {% tab title="Python" %} ```python import requests from pprint import pprint # Obtenha a resposta do endpoint de IPs de callback. response = requests.request( method='GET', url='https://data.oxylabs.io/v1/info/callbacker_ips', auth=('user', 'pass1'), ) # Print prettified JSON response to stdout. pprint(response.json()) ``` {% endtab %} {% tab title="PHP" %} ```php ``` {% endtab %} {% tab title="C#" %} ```csharp using System; using System.Net.Http; using System.Threading.Tasks; namespace OxyApi { class Program { static async Task Main() { const string Username = "YOUR_USERNAME"; const string Password = "YOUR_PASSWORD"; var client = new HttpClient(); Uri baseUri = new Uri("https://data.oxylabs.io"); client.BaseAddress = baseUri; var requestMessage = new HttpRequestMessage(HttpMethod.Get, "/v1/info/callbacker_ips"); var authenticationString = $"{Username}:{Password}"; var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.ASCIIEncoding.UTF8.GetBytes(authenticationString)); requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString); var response = await client.SendAsync(requestMessage); var contents = await response.Content.ReadAsStringAsync(); Console.WriteLine(contents); } } } ``` {% endtab %} {% tab title="Golang" %} ```go package main import ( "fmt" "io/ioutil" "net/http" ) func main() { const Username = "YOUR_USERNAME" const Password = "YOUR_PASSWORD" client := &http.Client{} request, _ := http.NewRequest("GET", "https://data.oxylabs.io/v1/info/callbacker_ips", nil, ) request.Header.Add("Content-type", "application/json") request.SetBasicAuth(Username, Password) response, _ := client.Do(request) responseText, _ := ioutil.ReadAll(response.Body) fmt.Println(string(responseText)) } ``` {% endtab %} {% tab title="Java" %} ```java package org.example; import okhttp3.*; public class Main implements Runnable { private static final String AUTHORIZATION_HEADER = "Authorization"; public static final String USERNAME = "YOUR_USERNAME"; public static final String PASSWORD = "YOUR_PASSWORD"; public void run() { Authenticator authenticator = (route, response) -> { String credential = Credentials.basic(USERNAME, PASSWORD); return response .request() .newBuilder() .header(AUTHORIZATION_HEADER, credential) .build(); }; var client = new OkHttpClient.Builder() .authenticator(authenticator) .build(); var request = new Request.Builder() .url("https://data.oxylabs.io/v1/info/callbacker_ips") .get() .build(); try (var response = client.newCall(request).execute()) { assert response.body() != null; System.out.println(response.body().string()); } catch (Exception exception) { System.out.println("Error: " + exception.getMessage()); } System.exit(0); } public static void main(String[] args) { new Thread(new Main()).start(); } } ``` {% endtab %} {% tab title="Node.js" %} ```shell import fetch from 'node-fetch'; const username = 'YOUR_USERNAME'; const password = 'YOUR_PASSWORD'; const response = await fetch('https://data.oxylabs.io/v1/info/callbacker_ips', { method: 'get', headers: { 'Content-Type': 'application/json', 'Authorization': 'Basic ' + Buffer.from(`${username}:${password}`).toString('base64'), } }); console.log(await response.json()); ``` {% endtab %} {% endtabs %} ### Saída A API retornará a lista de IPs que estão fazendo solicitações de callback para o seu sistema: ```json { "ips": [ "x.x.x.x", "y.y.y.y" ] } ``` ## Scheduler Scheduler é um serviço que você pode usar para agendar jobs de scraping recorrentes. Ele amplia a funcionalidade da integração Push-Pull e é melhor usado junto com a [**integração Cloud**](/products/pt-br/web-scraper-api/features/result-processing-and-storage/cloud-storage.md) funcionalidade. Visite esta página para aprender como usar o recurso Scheduler: {% content-ref url="/pages/6f5b6ae8a853bbb5d2fc0353c97d5a2df37b1ca6" %} [Scheduler](/products/pt-br/web-scraper-api/features/scheduler.md) {% endcontent-ref %} --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://developers.oxylabs.io/products/pt-br/web-scraper-api/integration-methods/push-pull.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.