AI Overviews

The google_search source, together with render parameter set to html, is designed to retrieve AI Overview (formerly known as Search Generative Experience (SGE)) results.

Explore data dictionary for some of the AI Overview SERP features.

Request samples

curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "google_search",
        "query": "Emporio Armani EA3192",
        "render": "html"
    }'

import requests
from pprint import pprint

# Updated payload.
payload = {
    'source': 'google_search',
    'query': 'Emporio Armani EA3192',
    'render': 'html',
}

# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('user', 'pass1'),
    json=payload,
)

# Print prettified response to stdout.
pprint(response.json())

<?php

$params = [
    'source' => 'google_search',
    'query' => 'Emporio Armani EA3192',
    'render' => 'html',
];

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");

$headers = [];
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);

https://realtime.oxylabs.io/v1/queries?source=google_search&query=Emporio%20Armani%20EA3192&render=html&access_token=12345abcde

{
    "source": "google_search",
    "query": "Emporio Armani EA3192",
    "render": "html"
}

We use synchronous Realtime integration method in our examples. If you would like to use Proxy Endpoint or asynchronous Push-Pull integration, refer to the integration methods section.

Request parameter values

Generic

Basic setup and customization options for scraping Google Web search results.

Parameter

Description

Default Value

source

Sets the scraper.

google_search

query

UTF-encoded keyword.

render

Enables JavaScript rendering when set to html. More info.

parse

Returns parsed data when set to true. Explore output data dictionary.

false

callback_url

URL to your callback endpoint. More info.

user_agent_type

Device type and browser. The full list can be found here.

desktop

- mandatory parameter

Google Advanced Search Operators

When scraping, you might find it useful to combine Google advanced search operators with your query. It enables you to customize the scope of the search, ensuring that the results are more relevant and focused. Explore these special commands here and here. See an example below.

{
    "source": "google_search",
    "query": "iphone 15 launch inurl:apple",
}

Localization

Adapt search results to specific geographical locations, domains, and languages.

Parameter

Description

Default Value

geo_location

The geographical location that the result should be adapted for. Using this parameter correctly is extremely important to get the right data. For more information, read about our suggested geo_location parameter structures here.

domain

Domain localization for Google. The full list of available domains can be found here.

com

locale

Accept-Language header value which changes your Google search page web interface language. More info.

context: results_language

Results language. List of supported Google languages can be found here.

Pagination

Controls for managing the pagination and retrieval of search results.

Parameter

Description

Default Value

start_page

Starting page number.

1

pages

Number of pages to retrieve.

1

limit

Number of results to retrieve in each page.

10

context:limit_per_page

Scrape multiple pages using the same IP address and session (cookie set). By specifying the page numbers in a JSON array with the page key and indicating the number of organic results per page using the limit key, you can minimize the chance of seeing overlapping organic results across pages (e.g., the last organic result on the first page being the same as the first organic result on the second page). See example.

Continuous scroll support

The SERP Scraper API fully supports Google Search continuous scroll. It automatically detects continuous scrolling layouts, efficiently loading the requested organic results without any extra parameters required.

Limit per page

To use this feature, include a JSON array with JSON objects containing the following data:

Parameter

Description

Example

page

The number of the page you would like to scrape. Any integer value greater than 0 will work

1

limit

The number of results on the page in question. Any integer value between 1 and 100 (inclusive) will work.

90

Request sample

{
    "source": "google_search",
    "query": "adidas",
    "parse": true,
    "context": [
        {
            "key": "limit_per_page",
            "value": [
                {"page": 1, "limit": 10},
                {"page": 2, "limit": 90}
                    ]
        }]
}

Filtering

Options to filter and refine search results based on various criteria.

Parameter

Description

Default Value

context:filter

Setting the value of this parameter to 0 lets you see results that would otherwise be excluded due to similarity to other results.

1

context:safe_search

Safe search. Set to true to enable it.

false

context: tbm

To-be-matched or tbm parameter. Accepted values are: app, blg, bks, dsc, isch, nws, pts, plcs, rcp, lcl

context: tbs

tbs parameter. This parameter is like a container for more obscure google parameters, like limiting/sorting results by date as well as other filters some of which depend on the tbm parameter (e.g. tbs=app_os:1 is only available with tbm value app). More info here.

Other

Additional advanced settings and controls for specialized requirements.

Parameter

Description

Default Value

context: fpstate

Setting the fpstate value to aig will make Google load more apps. This parameter is only useful if used together with the render parameter.

context: nfpr

true will turn off spelling auto-correction

false

Structured data

Data dictionary

All of the available keys within AI Overview SERP feature are defined in the table below:

Key (results.ai_overview)

Description

Type

answer_text

General information about the Google Search query term.

array

bullet_list

Indicates information on items rendered in a bulleted list within the AI Overview SERP feature.

array

products

A list of products with their respective details.

object

source_panel

A list of sources used to generate AI Overview result in Google SERP feature.

object

additional_questions

A list of additional questions with their respective details.

object

recipes

A list of recipes with their respective details.

object

info_list

A list of information most frequently used to represent steps of a process relevant to the search query.

array

Explore examples of AI Overview results below, categorized by different search queries: Item, Recipes, Instruction.

Item

In the following example, "Emporio Armani EA3192" query was used.

...
  "ai_overview": {
      "answer_text": [
        {
          "text": [
            "Emporio Armani is a luxury brand known for its timeless designs and commitment to Italian craftsmanship.",
            "The Emporio Armani EA3192 is a pair of prescription glasses that are designed for men.",
            "They have a rectangular frame that is made from hypoallergenic, lightweight plastic.",
            "The lenses are UV and anti-reflective.",
            "Here are some insights from reviews:"
          ],
          "pos": 1,
          "pos_overall": 1
        }
      ],
      "bullet_list": [
        {
          "list_title": "Pros",
          "points": [
            "Comfortable",
            "Minimal branding"
          ],
          "pos": 1,
          "pos_overall": 2
        }
      ],
      "products": {
        "items": [
          {
            "product_name": "Emporio Armani Men Eyeglasses Shiny Black Size: 53",
            "price": "$163.00",
            "previous_price": null,
            "seller": "Lenscrafters",
            "shipping": "Free delivery",
            "url": "https://www.lenscrafters.com/lc-us/emporio-armani/8056597975810?srsltid=AfmBOoqWO55KIJevhQp_c5OAp0JOM_8Uza3_gCvn9S4x4Vz342AUSQewzJI",
            "pos": 1
          },
	...
        ],
        "pos_overall": 3
      },
      "source_panel": {
        "items": [
          {
            "url": "https://www.italist.com/magazine/what-is-emporio-armani/#:~:text=Emporio%20Armani%20is%20a%20luxury,reinforce%20the%20perception%20of%20luxury.",
            "source": "italist.com",
            "date": "May 5, 2023",
            "title": "What is Emporio Armani and is Emporio Armani a luxury brand?",
            "pos": 1
          },
        ...
        ],
        "pos_overall": 1
      },
      "additional_questions": {
        "items": [
          "Is Emporio Armani the same as Giorgio Armani?",
          "Emporio Armani EA3178",
          "Emporio Armani EA 3098"
        ],
        "pos_overall": 4
      }
    }
  },
...

Answer text

Key (results.ai_overview.answer_text)

Description

Type

text

A short answer on the Google Search query.

array

pos

An indicator denoting the position of a given item among all AI Overview answer results.

integer

pos_overall

An indication of the position of the AI Overview Answer result within the AI Overview SERP feature.

integer

Bullet list

Key (results.ai_overview.bullet_list)

Description

Type

list_title

The title of the bullet list.

string

points

The items rendered in the bullet list. All items are comma-separated.

array

pos

An indicator denoting the position of a given item among all AI Overview bullet list results.

integer

pos_overall

An indication of the position of the AI Overview bullet list results within the AI Overview SERP feature.

integer

Products

Key (results.ai_overview.products)

Description

Type

items

A list of products with their respective details.

array

items.product_name

The name of the product.

string

items.price

The current price of the product.

string

items.previous_price

The initial price of the product (if applicable).

string

items.seller

The seller of the product.

string

items.shipping

The type of shipping available for the product.

string

items.url

The URL of the product.

string

items.pos

An indicator denoting the position of a given item among all AI Overview Product results.

integer

pos_overall

An indication of the position of the AI Overview Products result within the AI Overview feature.

integer

Source panel

Key (results.ai_overview.source_panel)

Description

Type

items

A list of sources used to generate AI Overview result in Google SERP feature.

object

items.url

The URL of the website.

string

items.source

The name of the website.

string

items.date

The date when the article has been published.

string

items.title

The title of the article.

string

items.pos

An indicator denoting the position of a given item among all AI Overview source panel results.

integer

pos_overall

An indication of the position of the AI Overview Source panel result within the AI Overview feature.

integer

Additional questions

Key (results.ai_overview.additional_questions)

Description

Type

additional_questions.items

A list of comma-separated Additional questions relevant to the original search query.

array

additional_questions.pos_overall

An indication of the position of the AI Overview Additional questions result within the AI Overview feature.

integer

Recipe

In the following example, "recipe for an apple pie" query was used.

...  
 "ai_overview": {
    ...
      "recipes": {
        "items": [
          {
            "description": "This simple recipe features a homemade crust, Granny Smith apples, and warm spices.",
            "image": "data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/...",
            "source": "Allrecipes",
            "rating": "4.8",
            "rating_count": "13K",
            "duration": "90 min",
            "url": "https://www.allrecipes.com/recipe/12682/apple-pie-by-grandma-ople/",
            "title": "Apple Pie by Grandma Ople"
          },
...
          ],
        "pos_overall": 2
      },
...

Recipes

Key (results.ai_overview.recipes)

Description

Type

items

A list of recipes with their respective details.

array

items.description

A short description of the recipe.

string

items.image

A base64-encoded thumbnail image of the recipe.

string

items.source

The name of the site where the recipe is published.

string

items.rating

The rating of the recipe.

string

items.rating_count

The number of ratings.

string

items.duration

The time that is needed to prepare the dish.

string

items.url

The URL to the full recipe.

string

items.title

The title of the recipe.

string

pos_overall

An indication of the position of the Recipes result within the AI Overview SERP feature.

integer

In the following example, "recipe of a juicy succulent steak" query was used.

Info list

Key (results.ai_overview.info_list)

Description

Type

list_title

The title of the list.

string

list_items

Contains a list of items with their respective details.

array

list_items.title

The title of the section within the list.

string

list_items.content

A short description of the section.

string

list_item.pos

An indicator denoting the position of a given item among all AI Overview Info list results.

integer

list_items.pos_overall

An indicator denoting the position of a given item among all AI Overview Info list results.

integer

Instruction

In the following example, "docker exec commands explained" query was used.

...
  "ai_overview": {
      "answer_text": [
        {
          "text": [
            "The docker execExecIn computing, exec is a functionality of an operating system that runs an executable file in the context of an already existing process, replacing the previous executable...
            "It is a powerful tool that can be used for a variety of tasks, such as troubleshooting a container, running a script, or installing software."
          ],
          "pos": 1,
          "pos_overall": 1
        },
        {
          "text": [
            "The syntax for the docker exec command is as follows:"
          ],
          "command": [
            "docker exec [OPTIONS] CONTAINER COMMAND [ARG...]"
          ],
          "pos": 2,
          "pos_overall": 2
        },
        ...
        }
      ],
...

Answer text command

Note: details are rendered in the same array as for item related search query, however, one new type of key (command) appears for instruction related type of query

Key (results.ai_overview.answer_text)

Description

Type

command

An instruction given to a computer or software to perform a specific task.

string

PreviousWeb Search NextImage Search

Last updated 4 months ago