News Search

The google_search source is designed to retrieve Google Search results (SERPs). This sub-page specifically presents data related to Google News Search. To explore other result types, read here: Web Search, Image Search.

To scrape Google News search, you must include the context:tbm parameter with the value set to nws.

Explore data dictionary for each News SERP feature, offering a brief description, screenshot, parsed JSON code snippet, and a table defining each parsed field. Navigate through the details using the right-side navigation or scrolling down the page.

Query parameters

Parameter Description Default Value

Parameter	Description	Default Value
source	Data source. More info.	`google_search`
`domain`	Domain localization for Google. The full list of available domains can be found here.	`com`
query	UTF-encoded keyword.	-
`start_page`	Starting page number.	`1`
`pages`	Number of pages to retrieve.	`1`
`limit`	Number of results to retrieve in each page.	`10`
`locale`	`Accept-Language` header value which changes your Google search page web interface language. More info.	-
`geo_location`	The geographical location that the result should be adapted for. Using this parameter correctly is extremely important to get the right data. For more information, read about our suggested `geo_location` parameter structures here.	-
`user_agent_type`	Device type and browser. The full list can be found here.	`desktop`
`render`	Enables JavaScript rendering. More info.
`callback_url`	URL to your callback endpoint. More info.	-
`parse`	`true` will return parsed data.	-
context: tbm	It stands for `to-be-matched`. To get News search results, set value to `nws`.	-
`context`: `limit_per_page`	If you want to scrape multiple pages with the same IP address, include a JSON array and specify the page numbers using the `page` key. You must also indicate the number of organic results on each page by adding a `limit` key. See example.	-
`context`: `nfpr`	`true` will turn off spelling auto-correction	`false`
`context`: `safe_search`	Safe search. Set to `true` to enable it.	`false`
`context`: `tbs`	`tbs` parameter acts as a container for Google parameters, including options for limiting or sorting results by date, among other filters. Some of these filters are dependent on the `tbm` parameter; for instance, `tbs=app_os:1` is only available when the `tbm` value is set to `app`. More info here.	-

source

Data source. More info.

google_search

domain

Domain localization for Google. The full list of available domains can be found here.

com

query

UTF-encoded keyword.

start_page

Starting page number.

1

pages

Number of pages to retrieve.

1

limit

Number of results to retrieve in each page.

10

locale

Accept-Language header value which changes your Google search page web interface language. More info.

geo_location

The geographical location that the result should be adapted for. Using this parameter correctly is extremely important to get the right data. For more information, read about our suggested geo_location parameter structures here.

user_agent_type

Device type and browser. The full list can be found here.

desktop

render

Enables JavaScript rendering. More info.

callback_url

URL to your callback endpoint. More info.

parse

true will return parsed data.

context: tbm

It stands for to-be-matched. To get News search results, set value to nws.

context:

limit_per_page

If you want to scrape multiple pages with the same IP address, include a JSON array and specify the page numbers using the page key. You must also indicate the number of organic results on each page by adding a limit key. See example.

context: nfpr

true will turn off spelling auto-correction

false

context:

safe_search

Safe search. Set to true to enable it.

false

context: tbs

tbs parameter acts as a container for Google parameters, including options for limiting or sorting results by date, among other filters. Some of these filters are dependent on the tbm parameter; for instance, tbs=app_os:1 is only available when the tbm value is set to app. More info here.

- required parameter

Google Advanced Search Operators

When scraping, you might find it useful to combine Google advanced search operators with your query. It enables you to customize the scope of the search, ensuring that the results are more relevant and focused. Explore these special commands here and here. See an example below.

{
    "source": "google_search",
    "query": "iphone 15 launch inurl:apple",
}

Code examples

In the example below, we make a request to obtain News search result pages for the search term adidas on the google.nl domain.

{
    "source": "google_search",
    "domain": "nl",
    "query": "adidas",
    "parse": true,
    "context": [
        {
            "key": "tbm",
            "value": "nws"
        }]
}

curl --user "user:pass1" 'https://realtime.oxylabs.io/v1/queries' -H "Content-Type: application/json" -d '{"source": "google_search", "domain": "nl", "query": "adidas", "parse": true, "context": [{"key": "tbm", "value": "nws"}]}'

import requests
from pprint import pprint

# Structure payload.
payload = {
    'source': 'google_search',
    'domain': 'nl',
    'query': 'adidas',
    'parse': True,
    'context': [
        {'key': 'tbm', 'value': 'nws'},
    ],
}

# Get response.
response = requests.post(
    'https://realtime.oxylabs.io/v1/queries',
    auth=('user', 'pass1'),
    json=payload,
)

# Print prettified response to stdout.
pprint(response.json())

<?php

$params = [
    'source' => 'google_search',
    'domain' => 'nl',
    'query' => 'adidas',
    'parse' => true,
    'context' => [
        [
            'key' => 'tbm',
            'value' => 'nws',
        ],
    ],
];

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "user" . ":" . "pass1");

$headers = [];
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}

curl_close($ch);

source=google_search&domain=nl&query=adidas&parse=true&context[0][key]=tbm&context[0][value]=nws&access_token=12345abcde

The example above uses the Realtime integration method. If you would like to use some other integration method in your query (e.g. Push-Pull or Proxy Endpoint), refer to the integration methods section.

Structured data

SERP Scraper API is capable of extracting either an HTML or JSON object that contains Google search results, offering structured data on various elements of the results page.

google_search news structured output

{
    "results": [
        {
            "content": {
                "url": "https://www.google.com/search?q=adidas&tbm=nws&uule=w+CAIQICINdW5pdGVkIHN0YXRlcw&gl=us&hl=en",
                "page": 1,
                "results": {
                    "main": [
                        {
                            "url": "https://www.cnn.com/2022/05/06/business/under-armour-stock-adidas-nike/index.html",
                            "desc": "Tripped-up supply chains and a coronavirus surge in China are causing \nheadaches for top athletic brands.",
                            "title": "Wall Street is fed up with Under Armour, Nike and Adidas",
                            "source": "CNN",
                            "pos_overall": 1,
                            "relative_publish_date": "2 days ago"
                        },
                        ...
                        {
                            "url": "https://www.cnbc.com/2022/05/06/dsw-tests-layout-to-spotlight-brands-like-adidas-crocs-birkenstock.html",
                            "desc": "DSW is trying out a new store look and layout at a location opening this \nweekend in Houston, in an attempt to focus customers' attention on...",
                            "title": "DSW is testing a store layout that puts the spotlight on brands like \nAdidas, Crocs and Birkenstock",
                            "source": "CNBC",
                            "pos_overall": 10,
                            "relative_publish_date": "2 days ago"
                        }
                    ],
                    "total_results_count": 57300000
                },
                "parse_status_code": 12000
            },
            "created_at": "2022-05-09 07:25:03",
            "updated_at": "2022-05-09 07:25:07",
            "page": 1,
            "url": "https://www.google.com/search?q=adidas&tbm=nws&uule=w+CAIQICINdW5pdGVkIHN0YXRlcw&gl=us&hl=en",
            "job_id": "6929330379711060993",
            "status_code": 200,
            "parser_type": "v2"
        }
    ]
}

We only parse news search results for desktop searches.

Data dictionary

HTML example

JSON structure

The Google News Search structured output includes fields like URL, page, results, and others. The table below presents a detailed list of each SERP feature we parse, along with its description and data type. The table also includes some metadata.

The number of items and fields for a specific result type may vary depending on the search query.

Key Description Type

Key	Description	Type
`url`	The URL of the Google search page.	string
`results`	A dictionary containing the results of the search.	array
`results.main`	A list of unpaid news results with their respective details.	array
`results.additional`	A list of trending articles with their respective details.	object
`results.total_results_count`	The total number of results found for the search query.	array
`parse_status_code`	The status code of the parsing job. You can see the parser status codes described here.	integer
`created_at`	The timestamp when the scraping job was created.	timestamp
`updated_at`	The timestamp when the scraping job was finished.	timestamp
`page`	Page number relative to the Google SERP pagination.	integer
`job_id`	The ID of the job associated with the scraping job.	string
`status_code`	The status code of the scraping job. You can see the scraper status codes described here.	integer

url

The URL of the Google search page.

string

results

A dictionary containing the results of the search.

array

results.main

A list of unpaid news results with their respective details.

array

results.additional

A list of trending articles with their respective details.

object

results.total_results_count

The total number of results found for the search query.

array

parse_status_code

The status code of the parsing job. You can see the parser status codes described here.

integer

created_at

The timestamp when the scraping job was created.

timestamp

updated_at

The timestamp when the scraping job was finished.

timestamp

page

Page number relative to the Google SERP pagination.

integer

job_id

The ID of the job associated with the scraping job.

string

status_code

The status code of the scraping job. You can see the scraper status codes described here.

integer

In the following sections, parsed JSON code snippets are shortened where more than one item for the result type is available.

Main

Displays a list of unpaid news results, providing relevant details for each article.

...
"main": [
    {
        "url": "https://www.yahoo.com/lifestyle/tiger-woods-nikes-epic-partnership-015311819.html",
        "desc": "That there could ever be a world in which Tiger Woods wasn't sponsored by \nNike seemed...",
        "title": "How Tiger Woods and Nike's Epic Partnership Fell Apart",
        "source": "Yahoo",
        "pos_overall": 1,
        "relative_publish_date": "1 day ago"
    },
                       ...
},

...

Key (results.main) Description Type

Key (results.main)	Description	Type
`url`	The URL to the full article.	string
`desc`	A short excerpt from the full article.	string
`title`	The title of the article.	string
`source`	The name of the website where the article is published.	string
`pos_overall`	Indicates the overall position of the result within the main results of News SERP.	integer
`relative_publish_date`	Describes how long ago the article was published.	string

url

The URL to the full article.

string

desc

A short excerpt from the full article.

string

title

The title of the article.

string

source

The name of the website where the article is published.

string

pos_overall

Indicates the overall position of the result within the main results of News SERP.

integer

relative_publish_date

Describes how long ago the article was published.

string

Additional

Presents a list of trending articles, accompanied by relevant details.

...
"additional": [
    {
        "items": [
            {
                "pos": 1,
                "url": "https://www.complex.com/sneakers/a/brendan-dunne/nike-book-1-colorways-haven-hike-rattlesnake",
                "title": "Nike Book 1 Colorways Haven Hike Rattlesnake",
                "source": "Complex",
                "relative_publish_date": "1 day ago"
            },
         ...
        ],
        "pos_overall": 2,
        "section_title": "Devin Booker confirms issues with Nike Book 1 launch"
    }
...

Key (results.additional) Description Type

Key (results.additional)	Description	Type
`items`	A list of articles with their respective details.	array
`items.pos`	A unique indicator denoting the article position in the list.	integer
`items.url`	The URL to the full article.	string
`items.title`	The title of the article.	string
`items.source`	The name of the website where the article is published.	string
`items.relative_publish_date`	Describes how long ago the article was published.	string
`pos_overall`	Indicates the overall position of the result within the additional results of News SERP.	integer
`section_title`	The name of the additional section.	string

items

A list of articles with their respective details.

array

items.pos

A unique indicator denoting the article position in the list.

integer

items.url

The URL to the full article.

string

items.title

The title of the article.

string

items.source

The name of the website where the article is published.

string

items.relative_publish_date

Describes how long ago the article was published.

string

pos_overall

Indicates the overall position of the result within the additional results of News SERP.

integer

section_title

The name of the additional section.

string

PreviousImage Search NextAds

Last updated 1 month ago