AI Overviews

The google_search source, together with render parameter set to html, is designed to retrieve AI Overview (formerly known as Search Generative Experience (SGE)) results.

Explore data dictionary for some of the AI Overview SERP features.

Request samples

curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "google_search",
        "query": "Emporio Armani EA3192",
        "render": "html"
    }'

import requests
from pprint import pprint

# Updated payload.
payload = {
    'source': 'google_search',
    'query': 'Emporio Armani EA3192',
    'render': 'html',
}

# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('user', 'pass1'),
    json=payload,
)

# Print prettified response to stdout.
pprint(response.json())

<?php

$params = [
    'source' => 'google_search',
    'query' => 'Emporio Armani EA3192',
    'render' => 'html',
];

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");

$headers = [];
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);

https://realtime.oxylabs.io/v1/queries?source=google_search&query=Emporio%20Armani%20EA3192&render=html&access_token=12345abcde

{
    "source": "google_search",
    "query": "Emporio Armani EA3192",
    "render": "html"
}

We use synchronous Realtime integration method in our examples. If you would like to use Proxy Endpoint or asynchronous Push-Pull integration, refer to the integration methods section.

Request parameter values

Generic

Basic setup and customization options for scraping Google Web search results.

- mandatory parameter

Google Advanced Search Operators

When scraping, you might find it useful to combine Google advanced search operators with your query. It enables you to customize the scope of the search, ensuring that the results are more relevant and focused. Explore these special commands here and here. See an example below.

{
    "source": "google_search",
    "query": "iphone 15 launch inurl:apple",
}

Localization

Adapt search results to specific geographical locations, domains, and languages.

Pagination

Controls for managing the pagination and retrieval of search results.

Continuous scroll support

The SERP Scraper API fully supports Google Search continuous scroll. It automatically detects continuous scrolling layouts, efficiently loading the requested organic results without any extra parameters required.

Limit per page

To use this feature, include a JSON array with JSON objects containing the following data:

Request sample

{
    "source": "google_search",
    "query": "adidas",
    "parse": true,
    "context": [
        {
            "key": "limit_per_page",
            "value": [
                {"page": 1, "limit": 10},
                {"page": 2, "limit": 90}
                    ]
        }]
}

Filtering

Options to filter and refine search results based on various criteria.

Other

Additional advanced settings and controls for specialized requirements.

Structured data

Data dictionary

All of the available keys within AI Overview SERP feature are defined in the table below:

Explore examples of AI Overview results below, categorized by different search queries: Item, Recipes, Instruction.

Item

In the following example, "Emporio Armani EA3192" query was used.

...
  "ai_overview": {
      "answer_text": [
        {
          "text": [
            "Emporio Armani is a luxury brand known for its timeless designs and commitment to Italian craftsmanship.",
            "The Emporio Armani EA3192 is a pair of prescription glasses that are designed for men.",
            "They have a rectangular frame that is made from hypoallergenic, lightweight plastic.",
            "The lenses are UV and anti-reflective.",
            "Here are some insights from reviews:"
          ],
          "pos": 1,
          "pos_overall": 1
        }
      ],
      "bullet_list": [
        {
          "list_title": "Pros",
          "points": [
            "Comfortable",
            "Minimal branding"
          ],
          "pos": 1,
          "pos_overall": 2
        }
      ],
      "products": {
        "items": [
          {
            "product_name": "Emporio Armani Men Eyeglasses Shiny Black Size: 53",
            "price": "$163.00",
            "previous_price": null,
            "seller": "Lenscrafters",
            "shipping": "Free delivery",
            "url": "https://www.lenscrafters.com/lc-us/emporio-armani/8056597975810?srsltid=AfmBOoqWO55KIJevhQp_c5OAp0JOM_8Uza3_gCvn9S4x4Vz342AUSQewzJI",
            "pos": 1
          },
	...
        ],
        "pos_overall": 3
      },
      "source_panel": {
        "items": [
          {
            "url": "https://www.italist.com/magazine/what-is-emporio-armani/#:~:text=Emporio%20Armani%20is%20a%20luxury,reinforce%20the%20perception%20of%20luxury.",
            "source": "italist.com",
            "date": "May 5, 2023",
            "title": "What is Emporio Armani and is Emporio Armani a luxury brand?",
            "pos": 1
          },
        ...
        ],
        "pos_overall": 1
      },
      "additional_questions": {
        "items": [
          "Is Emporio Armani the same as Giorgio Armani?",
          "Emporio Armani EA3178",
          "Emporio Armani EA 3098"
        ],
        "pos_overall": 4
      }
    }
  },
...

Answer text

Bullet list

Products

Source panel

Additional questions

Recipe

In the following example, "recipe for an apple pie" query was used.

...  
 "ai_overview": {
    ...
      "recipes": {
        "items": [
          {
            "description": "This simple recipe features a homemade crust, Granny Smith apples, and warm spices.",
            "image": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/...",
            "source": "Allrecipes",
            "rating": "4.8",
            "rating_count": "13K",
            "duration": "90 min",
            "url": "https://www.allrecipes.com/recipe/12682/apple-pie-by-grandma-ople/",
            "title": "Apple Pie by Grandma Ople"
          },
...
          ],
        "pos_overall": 2
      },
...

Recipes

In the following example, "recipe of a juicy succulent steak" query was used.

Info list

Instruction

In the following example, "docker exec commands explained" query was used.

...
  "ai_overview": {
      "answer_text": [
        {
          "text": [
            "The docker execExecIn computing, exec is a functionality of an operating system that runs an executable file in the context of an already existing process, replacing the previous executable...
            "It is a powerful tool that can be used for a variety of tasks, such as troubleshooting a container, running a script, or installing software."
          ],
          "pos": 1,
          "pos_overall": 1
        },
        {
          "text": [
            "The syntax for the docker exec command is as follows:"
          ],
          "command": [
            "docker exec [OPTIONS] CONTAINER COMMAND [ARG...]"
          ],
          "pos": 2,
          "pos_overall": 2
        },
        ...
        }
      ],
...

Answer text command

Note: details are rendered in the same array as for item related search query, however, one new type of key (command) appears for instruction related type of query

PreviousWeb Search NextImage Search

Last updated 15 days ago