Web Search
Comprehensive Google Search scraper with parsed data for organic results, ads, snippets, top stories, and more.
The google_search source is designed to retrieve Google Search (SERPs) and Google AI Overviews results. This sub-page specifically presents information related to Google Web Search. To explore other result types, read Image Search or News Search.
Request samples
In the samples below, we make a request to get 2 results pages, from number 11 to number 12 , for search term adidas. API will return parsed results.
curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
"source": "google_search",
"query": "adidas",
"start_page": 11,
"pages": 2,
"parse": true
}'import requests
from pprint import pprint
# Structure payload.
payload = {
'source': 'google_search',
'query': 'adidas',
'start_page': 11,
'pages': 2,
'parse': True,
}
# Get response.
response = requests.request(
'POST',
'https://realtime.oxylabs.io/v1/queries',
auth=('USERNAME', 'PASSWORD'),
json=payload,
)
# Print prettified response to stdout.
pprint(response.json())We use synchronous Realtime integration method in our examples. If you would like to use Proxy Endpoint or asynchronous Push-Pull integration, refer to the integration methods section.
Request parameter values
Generic
Basic setup and customization options for scraping Google Web search results.
source
Sets the scraper. Use google_search.
–
query
The keyword or phrase to search for.
–
- mandatory parameter
Google Advanced Search Operators
When scraping, you might find it useful to combine Google advanced search operators with your query. It enables you to customize the scope of the search, ensuring that the results are more relevant and focused. Explore these special commands here and here. See an example below.
Localization
Adapt search results to specific geographical locations, locales, and languages.
geo_location
The geographical location that the result should be adapted for. Using this parameter correctly is extremely important to get the right data. For more information, read about our suggested geo_location parameter structures here.
-
locale
Accept-Language header value which changes your Google search page web interface language. More info.
-
Pagination
Controls for managing the pagination and retrieval of search results.
start_page
Starting page number.
1
pages
Number of pages to retrieve.
1
limit
Number of results to retrieve in each page.
10
context:limit_per_page
Scrape multiple pages using the same IP address and session (cookie set). By specifying the page numbers in a JSON array with the page key and indicating the number of organic results per page using the limit key, you can minimize the chance of seeing overlapping organic results across pages (e.g., the last organic result on the first page being the same as the first organic result on the second page). See example.
-
Continuous scroll support
Web Scraper API fully supports Google Search continuous scroll. It automatically detects continuous scrolling layouts, efficiently loading the requested organic results without any extra parameters required.
Limit per page
Due to recent Google limit changes, we've adjusted Web Scraper API's behavior. Maximum results per page will match Google's organic output, which is typically 10 results.
To use this feature, include a JSON array with JSON objects containing the following data:
page
The number of the page you would like to scrape. Any integer value greater than 0 will work
1
limit
The number of results on the page in question. Any integer value between 1 and 100 (inclusive) will work.
90
Request sample
Filtering
Options to filter and refine search results based on various criteria. Learn how to use context parameters here.
context:
filter
Setting the value of this parameter to 0 lets you see results that would otherwise be excluded due to similarity to other results.
1
context:
safe_search
Safe search. Set to true to enable it.
false
context:
udm
udm parameter allows switching between different search tabs, such as images, places, or videos, to customize the type of results displayed. Find the accepted values here.
-
context:
tbm
To-be-matched or tbm parameter. Available values: bks (books), isch (images), nws (news), vid (videos), lcl (local) results.
-
context:
tbs
This parameter is like a container for more obscure google parameters, like limiting/sorting results by date as well as other filters some of which depend on the tbm parameter. More info here.
-
udm and tbm context parameters cannot be used together in a single scraping request; please select only one. Using both simultaneously may lead to conflicts or unexpected behavior.
Other
Additional advanced settings and controls for specialized requirements.
context:
fpstate
Setting the fpstate value to aig will make Google load more apps. This parameter is only useful if used together with the render parameter.
-
context:
nfpr
true will turn off spelling auto-correction
false
Context parameters
All context parameters should be added to the context array as objects with key and value pairs, e.g.:
Structured data
Web Scraper API is capable of extracting either an HTML or JSON object that contains Google search results, offering structured data on various elements of the results page.
To help you identify certain elements, refer to the data dictionaries for each type of Google search result, whether obtained from desktop or mobile searches.
Output data dictionary
Navigate through the details using the right-side navigation or scrolling down the page.
JSON structure
The Google Web Search structured output includes fields like URL, page, results, and others. The table below presents a detailed list of each SERP feature we parse, along with its description and data type. The table also includes some metadata.
url
The URL of the Google search page.
string
page
Page number relative to the Google SERP pagination.
integer
results
A dictionary containing the results of the search.
object
results.pla
A list of product listing ads with their respective details.
object
results.paid
A list of sponsored results with their respective details.
array
results.images
A list of image results with their respective details.
object
results.flights
A list of flights with their respective details.
object
results.organic
A list of unpaid listings with their respective details.
array
results.organic_videos
A list of organic videos with their respective details.
array
results.top_sights
A list of notable landmarks or attractions with their respective details.
array
results.jobs
A list of jobs listings with their respective details.
object
results.local_service_ads
A list of sponsored local service providers.
object
results.video_boxes
Video boxes in the SERP with their respective details.
array
results.recipes
A list of recipes with their respective details.
object
results.twitter
A list of Twitter (X) results with their respective details.
array
results.local_pack
A list containing local business listings relevant to the search.
object
results.item_carousel
A list of illustrative items for informational queries with their respective details.
object
results.videos
A list of videos with their respective details.
object
results.hotels
A list of hotels relevant to the location used in the search query.
object
results.apps
A list of applications with their respective details.
array
results.finance
A summary of financial data for companies, including stock prices, market cap, and other key metrics.
object
results.sports_games
Information on recent sports games, detailing scores, teams, game types, and highlights.
object
results.discussions_and_forums
A list of discussion threads and forum posts with URLs, titles, sources, and comment counts.
object
results.featured_snippet
A list of specific results retrieved from the organic part of SERPs.
array
results.top_stories
A list of articles with their respective details when a search query is identified as news-oriented.
object
results.popular_products
A list of unpaid and organic displays of Google Shopping product listings with their respective details.
object
results.related_searches
A list of one or more related searches blocks presented at different positions on the search page.
array
results.related_questions
A list of related interrogative search queries with their respective details.
object
results.what_people_are_saying
A list of discussion threads and forum posts with URLs, titles, sources, top comments, engagement level and timeframes.
array
results.search_information
A list of details for the submitted search query.
object
total_results_count
The total number of results found for the search query.
string
last_visible_page
Value identifying the maximum page number visible in the search query results page. (-1 when loading of more results is initiated by scrolling).
integer
parse_status_code
The status code of the parsing job. You can see the parser status codes described here.
integer
created_at
The timestamp when the scraping job was created.
timestamp
updated_at
The timestamp when the scraping job was finished.
timestamp
job_id
The ID of the job associated with the scraping job.
string
status_code
The status code of the scraping job. You can see the scraper status codes described here.
integer
parser_type
The type of the parser used for breaking down the HTML content.
string
Product Listing Ads (PLA)
The pla section of search results refers to Product Listing Ads, also known as Google Shopping Ads, prominently displayed at the top of the search page in a carousel format.
items
All PLAs available within the page.
array
Desktop and Mobile
items.pos
An indicator denoting the position of a given item among PLA results.
integer
Desktop and Mobile
items.title
The title of the product in the listing ad.
string
Desktop and Mobile
items.url
The URL of the product
string
Desktop and Mobile
items.price
The price of the product in the listing ad.
string
Desktop and Mobile
items.seller
The seller of the product in the listing ad.
string
Desktop and Mobile
items.url_image
The URL of the product image.
string
Desktop
items.image_data
The base64-encoded thumbnail image of the product.
string
Desktop and Mobile
pos_overall
An indication of the position of the PLA SERP feature within the SERP.
integer
Desktop and Mobile
Paid
The paid section refers to advertisements powered by Google Ads, usually appearing above organic (non-paid) results.
pos
An indicator denoting the position of a given item among paid results.
integer
Desktop and Mobile
url
The URL of the paid result.
string
Desktop and Mobile
desc
A short description of the paid result.
string
Desktop and Mobile
title
The title of the paid result.
string
Desktop and Mobile
data_rw
Redirect URL of the paid result.
string
Desktop and Mobile
sitelinks
An object containing information about internal links to a website that appears under the first search result on Google.
object
Desktop and Mobile
sitelinks.inline/expanded
A list of URLs with their respective details. The name of the key (`inline`/`expanded`) specifies the type of the sitelinks element.
array
Desktop and Mobile
sitelinks.inline.url
The URL of the linked site.
string
Desktop and Mobile
sitelinks.inline.title
The title of the linked site.
string
Desktop and Mobile
sitelinks.inline.description
A short description of the linked site.
string
Desktop
url_shown
The short-hand URL visible just below the description.
string
Desktop and Mobile
pos_overall
An indication of the position of the paid result within the SERP.
integer
Desktop and Mobile
Images
The images section refers to image results displayed in different sections on a SERP page, either as a dedicated section or at the top. It's important to note that the structure of scraped data from Google Search results may differ from that of scraped data from Google Image Search results.
items
A list of images with their respective details.
array
Desktop and Mobile
items.alt
The alt text of the image.
string
Desktop and Mobile
items.pos
A unique indicator denoting the image position in the list.
integer
Desktop and Mobile
items.url
The URL of the web page containing the image.
string
Desktop and Mobile
pos_overall
An indication of the position of the Images SERP feature within the SERP.
integer
Desktop and Mobile
Organic
The organic section refers to unpaid listings in organic Google search results, determined by relevance through Google's algorithm. These listings appear below paid ads on search pages.
Example 1
Example 2
pos
An indicator denoting the position of a given item among organic results.
integer
Desktop and Mobile
url
The URL of the organic result.
string
Desktop and Mobile
desc
A short description of the organic result.
string
Desktop and Mobile
title
The title of the organic result.
string
Desktop and Mobile
images
The base64-encoded thumbnail images of the organic result.
string
Desktop and Mobile
sitelinks
Object contains `expanded` or `inline` element with more details on the sitelinks.
object
Desktop and Mobile
sitelinks.expanded/inline
A list of sitelinks of the organic result. The name of the key (inline/expanded) specifies the type of the sitelinks element.
array
Desktop and Mobile
sitelinks.expanded.url
The URL of the linked site.
string
Desktop and Mobile
sitelinks.expanded.title
The title of the linked site.
string
Desktop and Mobile
url_shown
The short-hand URL visible just below the description.
string
Desktop and Mobile
pos_overall
An indication of the position of the Organic result within the SERP.
integer
Desktop and Mobile
favicon_text
The text or name associated with the favicon (website icon).
string
Desktop
rating
The average rating of the product or content.
float
Desktop
review_count
The number of reviews associated with the product or content.
integer
Desktop
additional_info
Contains details about pricing, availability, product ratings, delivery options, return policies.
list of strings
Desktop
Organic Videos
The results.organic_videos contains an array of video listings that are displayed organically in the search results, meaning they are not paid advertisements. Each video listing includes relevant details such as the video title, URL, description.
pos
An indicator denoting the position of a given item among Organic Videos results.
integer
Desktop
url
The URL of the video.
string
Desktop
desc
Short description of the video.
string
Desktop
title
The title of the video.
string
Desktop
pos_overall
An indication of the position of the Organic Video result within the SERP.
integer
Desktop
Top Sights
The results.top_sights contains a list of prominent landmarks or attractions typically associated with the search location or topic. Each item includes details such as the position in the list and title.
items
A list of top attractions with their respective details.
array
Desktop
items.pos
The position of the Top Sights result within the Top Sights SERP feature.
integer
Desktop
items.title
The title of the tourist attraction site.
string
Desktop
pos_overall
An indication of the position of the Top Sights result within the SERP.
integer
Desktop
Jobs
The results.jobs contains job listings extracted from the search results. Each listing includes details such as the URL directing to the job posting, job title, source, employer, and location. The listings are sourced from various job boards, career websites, and other platforms indexed by Google.
listings
An array of all jobs listings available within the page.
array
Desktop
listings.url
The URL to the full job listing.
string
Desktop
listings.title
The title of the job position.
string
Desktop
listings.source
The source which contains the original job posting.
string
Desktop
listings.employer
The organization hiring for the identified position.
string
Desktop
listings.location
The location for the position.
string
Desktop
pos_overall
An indication of the position of the Jobs SERP feature within the SERP.
integer
Desktop
Twitter (X)
The twitter feature frequently surfaces for brands, movies, musicians, and other celebrities, typically positioned directly under the search result for the brand's website. This feature offers a direct link to the brand's Twitter feed, showcasing a carousel of its most recent tweets.
pos
An indicator denoting the position of a given item among organic results.
integer
Desktop
url
The URL of the profile containing the Twitter (X) posts.
string
Desktop
items
A list of Twitter (X) posts with their respective details.
array
Desktop
items.pos
An indicator denoting the position of a given tweet within the Twitter (X) SERP feature.
string
Desktop
items.url
The URL of the Twitter (X) post.
string
Desktop
items.content
A short description denoting all of the text from the relevant Twitter (X) post.
string
Desktop
items.timeframe
Describes how long ago the tweet was created.
string
Desktop
title
The title of the Twitter (X) profile.
string
Desktop
pos_overall
An indication of the position of the Twitter result within the SERP.
integer
Desktop
Local Pack
The local_pack displays the top three local search results based on the user's location or specified search location, including a map and additional information.
items
A list of locations with their respective details.
array
Desktop and Mobile
items.cid
A unique identification number assigned to a specific business listing.
string
Desktop and Mobile
items.pos
The position of the local result within the local pack.
integer
Desktop and Mobile
items.title
The title of the local entity.
string
Desktop and Mobile
items.rating
The rating of the local entity.
integer
Desktop and Mobile
items.address
The address of the local entity.
string
Desktop and Mobile
items.rating_count
The number of ratings for the local entity.
integer
Desktop and Mobile
pos_overall
An indication of the position of the Local Pack result within the SERP.
integer
Desktop and Mobile
Top Stories
top_stories is a dedicated section surfacing in Google Search for news-oriented queries.
items
A list of selected news articles with their respective details.
array
Desktop and Mobile
items.pos
The position of the article within the Top Stories SERP feature.
integer
Desktop and Mobile
items.url
The URL to the full article.
string
Desktop and Mobile
items.title
The title of the article.
string
Desktop and Mobile
items.section
The name of the Top Stories section.
string
Mobile
items.source
The name of the site where the article is published.
string
Desktop and Mobile
items.timeframe
Describes how long ago the article was published.
string
Desktop
pos_overall
Indicates the overall position of the Top Stories SERP feature within the SERP.
integer
Desktop and Mobile
Popular Products
The popular_products SERP feature presents a carousel of various products determined by their popularity and relevance to the user's query. This differs from Product Listing Ads (PLAs), which are influenced by advertisers. It's worth noting that multiple popular_products carousels may appear on a single SERP page.
items
A list of popular products with their respective details.
array
Desktop and Mobile
items.pos
A unique indicator for the position of a given popular product within the Popular Products SERP feature.
integer
Desktop and Mobile
items.url
The URL of the product's Google Shopping page.
string
Desktop and Mobile
items.title
The product title.
string
Desktop and Mobile
items.token
Product token (used with google_shopping_product source for seller, pricing, product details).
string
Desktop and Mobile
items.rating
The rating of a given popular product.
string
Desktop and Mobile
items.sellers
A list of sellers offering the product, with their respective details.
array
Desktop and Mobile
items.sellers.pos
The position of the seller within the sellers list.
integer
Desktop and Mobile
items.sellers.price_str
The price of the product by the seller.
string
Desktop and Mobile
items.sellers.seller_name
The name of the product seller.
string
Desktop and Mobile
items.currency
The currency code of listed price.
string
Desktop and Mobile
items.image_data
The base64-encoded thumbnail image of the product.
string
Desktop and Mobile
items.description
Short description of the product seller.
string
Desktop and Mobile
items.rating_count
The number of ratings for the product.
integer
Desktop and Mobile
pos_overall
Indicates the overall position of the Popular Products SERP feature within the SERP.
integer
Desktop and Mobile
Related Searches
The related_searches SERP feature displays alternative search queries related to the initial search keyword, typically located at the bottom of the SERP.
pos_overall
Indicates the overall position of this related searches block within the search page.
integer
Desktop and Mobile
related_searches
A list of related search strings within this block.
array
Desktop and Mobile
Related Questions
The related_questions feature is positioned below the main search results, presenting a list of questions directly related to the user's original search query. By offering a curated set of related questions, users can explore different aspects of their topic, enhancing their overall search experience.
items
A list containing all of the related questions.
array
Desktop and Mobile
items.pos
A unique indicator denoting the position of any given related question within the Related Questions SERP feature.
integer
Desktop and Mobile
items.answer
The answer to the related question.
string