AI Overviews

Extract Google AI Overviews from search results with parsed data for answer text, bullet lists, source panels, and info lists.

The google_search and google_ads sources can return AI Overviews from Google Search results in all supported global regions. ai_overviews result array supports multiple AI blocks in a single response, including sections like “People also ask” and “Things to know.”

To maximize the chance of receiving fully rendered AI Overviews content, you must include the following parameters:

Parameter

Value

source

google_search or google_ads

render

html

user_agent_type

desktop or mobile

Using these parameters also allows you to receive both the general response and AI Overviews content with a single request.

Google dynamically generates this content or loads a cached version, so AI-generated answers may vary over time, even with identical parameters.

Explore data dictionary for some of the AI Overviews SERP features.

AI Overviews regional availability

Google AI Overviews are available in most countries apart from a few exceptions. Currently biggest excluded countries are:

France
Monaco
China
Iran
Sudan
Syria
Cuba
North Korea

Google AI Overviews feature is continuously rolling out with more countries included over time.

Request samples

Below are code examples using google_search Web Scraper API source to capture Google AI Overviews.

curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "google_search",
        "query": "how to sell on amazon",
        "render": "html",
        "user_agent_type": "desktop",
        "parse": true
        
    }'

import requests
from pprint import pprint

# Updated payload.
payload = {
    'source': 'google_search',
    'query': 'how to sell on amazon',
    'render': 'html',
    'user_agent_type': 'desktop',
}

# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('user', 'pass1'),
    json=payload,
)

# Print prettified response to stdout.
pprint(response.json())

<?php

$params = [
    'source' => 'google_search',
    'query' => 'how to sell on amazon',
    'render' => 'html',
    'user_agent_type' => 'desktop',
];

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");

$headers = [];
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);

https://realtime.oxylabs.io/v1/queries?source=google_search&query=Emporio%20Armani%20EA3192&render=html&user_agent_type=desktop&access_token=12345abcde

{
    "source": "google_search",
    "query": "how to sell on amazon",
    "render": "html",
    "user_agent_type": "desktop"
}

We use synchronous Realtime integration method in our examples. If you would like to use Proxy Endpoint or asynchronous Push-Pull integration, refer to the integration methods section.

Working with Multiple AI Overviews

Since AI Overviews are now returned as an array, you have to iterate through them:

// Processing all AI Overviews in a response
response.results.ai_overviews.forEach((aiOverview, index) => {
  console.log(`Processing AI Overview #${index + 1}`);
  // Process individual components
  if (aiOverview.answer_text) {
    // Handle answer text
  }
  if (aiOverview.bullet_list) {
    // Handle bullet lists
  }
  // And so on...
});

# Processing all AI Overviews in a response
for index, ai_overview in enumerate(response['results']['ai_overviews']):
    print(f"Processing AI Overview #{index + 1}")
    # Process individual components
    if 'answer_text' in ai_overview:
        answer_texts = ai_overview['answer_text']
        for answer in answer_texts:
            # Process each answer text
            if 'fragments' in answer:
                for text_item in answer['fragments']:
                    print(f"Answer text: {fragment['text']}")
    
    if 'bullet_list' in ai_overview:
        bullet_lists = ai_overview['bullet_list']
        for bullet_list in bullet_lists:
            if 'list_title' in bullet_list:
                print(f"List title: {bullet_list['list_title']}")
            if 'points' in bullet_list:
                for point in bullet_list['points']:
                    point_text = point['text']
                    print(f"- {point_text}")
    # Continue processing other elements...

<?php
// Processing all AI Overviews in a response
if (isset($response['results']['ai_overviews']) && is_array($response['results']['ai_overviews'])) {
    foreach ($response['results']['ai_overviews'] as $index => $aiOverview) {
        $indexNumber = $index + 1;
        echo "Processing AI Overview #{$indexNumber}\n";
        
        // Process answer text
        if (isset($aiOverview['answer_text']) && is_array($aiOverview['answer_text'])) {
            foreach ($aiOverview['answer_text'] as $answer) {
                if (isset($answer['fragments']) && is_array($answer['fragments'])) {
                    foreach ($answer['fragments'] as $fragment) {
                        $textItem = $fragment['text'];
                        echo "Answer text: {$textItem}\n";
                    }
                }
            }
        }
        
        // Process bullet lists
        if (isset($aiOverview['bullet_list']) && is_array($aiOverview['bullet_list'])) {
            foreach ($aiOverview['bullet_list'] as $bulletList) {
                if (isset($bulletList['list_title'])) {
                    echo "List title: {$bulletList['list_title']}\n";
                }
                if (isset($bulletList['points']) && is_array($bulletList['points'])) {
                    foreach ($bulletList['points'] as $point) {
                        $pointText = $point['text'];
                        echo "- {$pointText}\n";
                    }
                }
            }
        }
        // Continue processing other elements...
    }
}
?>

In most cases, there will be only one AI Overview, but your code should be prepared to handle multiple entries.

Request parameters

Generic

Basic setup and customization options for scraping Google Web search results with AI Overviews.

Parameter

Description

Default Value

source

Sets the scraper. Use google_search or google_ads.

google_search

query

The keyword or phrase to search for.

render

Enables JavaScript rendering when set to html. More info.

context: expand_aio

Set to true to expand Google AI Overviews (JavaScript rendering enabled).

false

parse

Returns parsed data when set to true.

false

callback_url

URL to your callback endpoint. More info

user_agent_type

Device type and browser. Use desktop or mobile.

desktop

- mandatory parameter

Google Advanced Search Operators

When scraping, you might find it useful to combine Google advanced search operators with your query. It enables you to customize the scope of the search, ensuring that the results are more relevant and focused. Explore these special commands here. See the example query below.

{
    "source": "google_search",
    "query": "iphone 15 launch inurl:apple", 
    "render": "html",
    "user_agent_type": "desktop"
}

Localization

Adapt search results to specific geographical locations and languages.

Parameter

Description

Default Value

geo_location

The geographical location that the result should be adapted for. Using this parameter correctly is extremely important to get the right data. For more information, read about our suggested geo_location parameter structures here.

locale

Accept-Language header value which changes your Google search page web interface language. More info.

Pagination

Controls for managing the pagination and retrieval of search results.

Parameter

Description

Default Value

start_page

Starting page number.

1

pages

Number of pages to retrieve.

1

limit

Number of results to retrieve in each page.

10

context:limit_per_page

Scrape multiple pages using the same IP address and session (cookie set). By specifying the page numbers in a JSON array with the page key and indicating the number of organic results per page using the limit key, you can minimize the chance of seeing overlapping organic results across pages (e.g., the last organic result on the first page being the same as the first organic result on the second page). See example.

Continuous scroll support

The Web Scraper API fully supports Google Search continuous scroll. It automatically detects continuous scrolling layouts, efficiently loading the requested organic results without any extra parameters required.

Limit per page

Due to recent Google limit changes, we've adjusted Web Scraper API's behavior. Maximum results per page will match Google's organic output, which is typically 10 results.

To use this feature, include a JSON array with JSON objects containing the following data:

Parameter

Description

Example

page

The number of the page you would like to scrape. Any integer value greater than 0 will work

1

limit

The number of results on the page in question. Any integer value between 1 and 100 (inclusive) will work.

90

Request sample

{
    "source": "google_search",
    "query": "how to sell on amazon",
    "render": "html",
    "user_agent_type": "dekstop",
    "parse": true,
    "context": [
        {
            "key": "limit_per_page",
            "value": [
                {"page": 1, "limit": 10},
                {"page": 2, "limit": 90}
                    ]
        }]
}

Filtering

Options to filter and refine search results based on various criteria. Learn how to use context parameters here.

Parameter

Description

Default Value

context: filter

Setting the value of this parameter to 0 lets you see results that would otherwise be excluded due to similarity to other results.

1

context: safe_search

Safe search. Set to true to enable it.

false

context: udm

udm parameter allows switching between different search tabs, such as images, places, or videos, to customize the type of results displayed. Find the accepted values here.

context: tbm

To-be-matched or tbm parameter. Accepted values are: app, blg, bks, dsc, isch, nws, pts, plcs, rcp, lcl

context: tbs

This parameter is like a container for more obscure Google parameters, like limiting/sorting results by date as well as other filters some of which depend on the tbm parameter (e.g. tbs=app_os:1 is only available with tbm value app). More info here.

udm and tbm context parameters cannot be used together in a single scraping request; please select only one. Using both simultaneously may lead to conflicts or unexpected behavior.

Other

Additional advanced settings and controls for specialized requirements.

Parameter

Description

Default Value

context: fpstate

Setting the fpstate value to aig will make Google load more apps. This parameter is only useful if used together with the render parameter.

context: nfpr

true will turn off spelling auto-correction

false

Context parameters

All context parameters should be added to the context array as objects with key and value pairs, e.g.:

...
"context": [
    {
        "key": "filter",
        "value": "0"
    }
]
...

Structured output

Data dictionary

All of the available keys within AI Overviews SERP feature are defined in the table below:

Key (results.ai_overviews)

Description

Type

answer_text

General information about the Google Search query term.

array

bullet_list

Indicates information on items rendered in a bulleted list within the AI Overviews SERP feature.

array

source_panel

A list of sources used to generate AI Overviews result in Google SERP feature.

object

info_list

A list of information most frequently used to represent steps of a process relevant to the search query.

array

pos_overall

An indication of the position of the AI Overviews SERP feature result among all available Google SERP results.

integer

General

In the following example, "Emporio Armani EA3192" query was used.

{
  "ai_overviews": [
    {
      "answer_text": [
        {
          "fragments": [
            {
              "text": "Emporio Armani is a luxury brand known for timeless designs.",
              "references": [
                {
                  "source": "http://italist.com ",
                  "url": "https://www.italist.com/magazine/what-is-emporio-armani/ "
                }
              ]
            }
          ],
          "pos": 1
        }
      ],
      "bullet_list": [
        {
          "list_title": "Pros",
          "points": [
            {"text": "Comfortable"},
            {"text": "Minimal branding"}
          ],
          "pos": 1
        }
      ],
      "source_panel": {
        "items": [
          {
            "url": "https://www.italist.com/...",
            "source": "http://italist.com ",
            "title": "What is Emporio Armani...",
            "pos": 1
          }
        ]
      },
      "pos_overall": 1
    }
  ]
}

Answer text

...
"answer_text": [
  {
    "fragments": [
      {
        "text": "Emporio Armani is a luxury brand known for timeless designs.",
        "references": [
          {
            "source": "http://italist.com",
            "url": "https://www.italist.com/magazine/what-is-emporio-armani/"
          }
        ]
      }
    ],
    "pos": 1
  }
]
...

Key (results.ai_overviews.answer_text)

Description

Type

fragments

A short answer to the Google Search query.

array

fragments.text

The answer text content of the fragment.

string

fragments.references

A list of sources referenced by the fragment.

array

fragments.references.source

The name of the referenced source.

string

fragments.references.url

The URL of the referenced source.

string

pos

An indicator denoting the position of a given item among all AI Overviews answer results.

integer

Bullet list

...
"bullet_list": [
  {
    "list_title": "Pros",
    "points": [
      {"text": "Comfortable"},
      {"text": "Minimal branding"}
    ],
    "pos": 1
  }
]
...

Key (results.ai_overviews.bullet_list)

Description

Type

list_title

The title of the bullet list.

string

points

The items rendered in the bullet list. All items are comma-separated.

array

points.text

The text content of the bullet point.

string

points.references

A list of sources referenced by the bullet point (optional).

array

pos

An indicator denoting the position of a given item among all AI Overviews bullet list results.

integer

Source panel

...
"source_panel": {
  "items": [
    {
      "url": "https://www.italist.com/...",
      "source": "http://italist.com",
      "title": "What is Emporio Armani...",
      "pos": 1
    }
  ]
}
...

Key (results.ai_overviews.source_panel)

Description

Type

items

A list of sources used to generate AI Overviews result in Google SERP feature.

object

items.url

The URL of the website.

string

items.source

The name of the website.

string

items.title

The title of the article.

string

items.pos

An indicator denoting the position of a given item among all AI Overviews source panel results.

integer

Info list

Key (results.ai_overviews.info_list)

Description

Type

list_title

The title of the list.

string

list_items

Contains a list of items with their respective details.

array

list_items.title

The title of the section within the list.

string

list_items.content

A short description of the section.

array

list_item.pos

An indicator denoting the position of a given item among all AI Overviews Info list results.

integer

Instructions

In the following example, "docker exec commands explained" query was used.

...
  "ai_overviews": [
    {
      "answer_text": [
        {
          "fragments": [
            "The docker execExecIn computing, exec is a functionality of an operating system that runs an executable file in the context of an already existing process, replacing the previous executable...",
            "It is a powerful tool that can be used for a variety of tasks, such as troubleshooting a container, running a script, or installing software."
          ],
          "pos": 1
        },
        {
          "fragments": [
            "The syntax for the docker exec command is as follows:"
          ],
          "pos": 2
        },
        ...
        }
      ]
    }
  ],
...

Answer text

"answer_text": [
  {
    "fragments": [
      "The docker exec...",
      "It is a powerful tool that can be used for a variety of tasks, such as troubleshooting a container, running a script, or installing software."
    ],
    "pos": 1
  }
]

Key (results.ai_overviews.answer_text)

Description

Type

fragments

Google Search query answer fragment in plain text.

array of strings

pos

An indicator denoting the position of the item in the Overview answer.

integer

PreviousAI Mode NextAds Max

Last updated 24 days ago

Was this helpful?