City Market

Learn how to start scraping City Market.

With Web Scraper API, you can scrape various types of City Market pages; below is an overview of all supported scrapers and their respective source values. The response returns the page HTML in content, in the standard response structure described in Realtime and Push-Pull.

For some targets our system forces JavaScript rendering even if you do not set render; those requests count as rendered traffic. The list of force-rendered targets and how to switch rendering off are in Forcing rendering on specific pages.

Source	Description
citymarket_search	Search page for a search term of your choice.
citymarket_product	Product page of a product ID of your choice.

Any other City Market page can be scraped by passing its url alone, no source needed.

Besides the parameters listed below, you can include additional parameters such as user_agent_type, geo_location, render, and more to customize your scraping request. Read more here.

Search

Request samples

cURL

curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "citymarket_search",
        "query": "milk"
    }'

Python

import requests
from pprint import pprint

# Structure payload.
payload = {
    'source': 'citymarket_search',
    'query': 'milk'
}

# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'),
    json=payload
)

# Instead of response with job status and results url, this will return the
# JSON response with the result.
pprint(response.json())

Node.js

const https = require("https");

const username = "USERNAME";
const password = "PASSWORD";
const body = {
    source: "citymarket_search",
    query: "milk"
};

const options = {
    hostname: "realtime.oxylabs.io",
    path: "/v1/queries",
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        Authorization:
            "Basic " + Buffer.from(`${username}:${password}`).toString("base64"),
    },
};

const request = https.request(options, (response) => {
    let data = "";

    response.on("data", (chunk) => {
        data += chunk;
    });

    response.on("end", () => {
        const responseData = JSON.parse(data);
        console.log(JSON.stringify(responseData, null, 2));
    });
});

request.on("error", (error) => {
    console.error("Error:", error);
});

request.write(JSON.stringify(body));
request.end();

HTTP

# The whole string you submit has to be URL-encoded.

https://realtime.oxylabs.io/v1/queries?source=citymarket_search&query=milk&access_token=12345abcde

PHP

<?php

$params = array(
    'source' => 'citymarket_search',
    'query' => 'milk'
);

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "USERNAME" . ":" . "PASSWORD");

$headers = array();
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);

Golang

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
)

func main() {
    const Username = "USERNAME"
    const Password = "PASSWORD"

    payload := map[string]interface{}{
        "source": "citymarket_search",
        "query": "milk",
    }

    jsonValue, _ := json.Marshal(payload)

    client := &http.Client{}
    request, _ := http.NewRequest("POST",
        "https://realtime.oxylabs.io/v1/queries",
        bytes.NewBuffer(jsonValue),
    )

    request.SetBasicAuth(Username, Password)
    response, _ := client.Do(request)

    responseText, _ := ioutil.ReadAll(response.Body)
    fmt.Println(string(responseText))
}

C#

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Net.Http.Json;
using System.Threading.Tasks;

namespace OxyApi
{
    class Program
    {
        static async Task Main()
        {
            const string Username = "USERNAME";
            const string Password = "PASSWORD";

            var parameters = new {
                source = "citymarket_search",
                query = "milk"
            };

            var client = new HttpClient();

            Uri baseUri = new Uri("https://realtime.oxylabs.io");
            client.BaseAddress = baseUri;

            var requestMessage = new HttpRequestMessage(HttpMethod.Post, "/v1/queries");
            requestMessage.Content = JsonContent.Create(parameters);

            var authenticationString = $"{Username}:{Password}";
            var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.ASCIIEncoding.UTF8.GetBytes(authenticationString));
            requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString);

            var response = await client.SendAsync(requestMessage);
            var contents = await response.Content.ReadAsStringAsync();

            Console.WriteLine(contents);
        }
    }
}

Java

package org.example;

import okhttp3.*;
import org.json.JSONObject;
import java.util.concurrent.TimeUnit;

public class Main implements Runnable {
    private static final String AUTHORIZATION_HEADER = "Authorization";
    public static final String USERNAME = "USERNAME";
    public static final String PASSWORD = "PASSWORD";

    public void run() {
        JSONObject jsonObject = new JSONObject();
        jsonObject.put("source", "citymarket_search");
        jsonObject.put("query", "milk");

        Authenticator authenticator = (route, response) -> {
            String credential = Credentials.basic(USERNAME, PASSWORD);
            return response
                    .request()
                    .newBuilder()
                    .header(AUTHORIZATION_HEADER, credential)
                    .build();
        };

        var client = new OkHttpClient.Builder()
                .authenticator(authenticator)
                .readTimeout(180, TimeUnit.SECONDS)
                .build();

        var mediaType = MediaType.parse("application/json; charset=utf-8");
        var body = RequestBody.create(jsonObject.toString(), mediaType);
        var request = new Request.Builder()
                .url("https://realtime.oxylabs.io/v1/queries")
                .post(body)
                .build();

        try (var response = client.newCall(request).execute()) {
            if (response.body() != null) {
                try (var responseBody = response.body()) {
                    System.out.println(responseBody.string());
                }
            }
        } catch (Exception exception) {
            System.out.println("Error: " + exception.getMessage());
        }

        System.exit(0);
    }

    public static void main(String[] args) {
        new Thread(new Main()).start();
    }
}

JSON

{
    "source": "citymarket_search",
    "query": "milk"
}

Request parameter values

Parameter	Description	Default Value
source	Sets the scraper.	citymarket_search
query	The keyword or phrase to search for products.	-
store_id, delivery_zip, fulfillment_type, price_range, brand	Optional localization and filter parameters, see Localization and Filtering.	-

- mandatory parameter

Product

Request samples

cURL

curl 'https://realtime.oxylabs.io/v1/queries' \
--user 'USERNAME:PASSWORD' \
-H 'Content-Type: application/json' \
-d '{
        "source": "citymarket_product",
        "product_id": "0001111016338"
    }'

Python

import requests
from pprint import pprint

# Structure payload.
payload = {
    'source': 'citymarket_product',
    'product_id': '0001111016338'
}

# Get response.
response = requests.request(
    'POST',
    'https://realtime.oxylabs.io/v1/queries',
    auth=('USERNAME', 'PASSWORD'),
    json=payload
)

# Instead of response with job status and results url, this will return the
# JSON response with the result.
pprint(response.json())

Node.js

const https = require("https");

const username = "USERNAME";
const password = "PASSWORD";
const body = {
    source: "citymarket_product",
    product_id: "0001111016338"
};

const options = {
    hostname: "realtime.oxylabs.io",
    path: "/v1/queries",
    method: "POST",
    headers: {
        "Content-Type": "application/json",
        Authorization:
            "Basic " + Buffer.from(`${username}:${password}`).toString("base64"),
    },
};

const request = https.request(options, (response) => {
    let data = "";

    response.on("data", (chunk) => {
        data += chunk;
    });

    response.on("end", () => {
        const responseData = JSON.parse(data);
        console.log(JSON.stringify(responseData, null, 2));
    });
});

request.on("error", (error) => {
    console.error("Error:", error);
});

request.write(JSON.stringify(body));
request.end();

HTTP

# The whole string you submit has to be URL-encoded.

https://realtime.oxylabs.io/v1/queries?source=citymarket_product&product_id=0001111016338&access_token=12345abcde

PHP

<?php

$params = array(
    'source' => 'citymarket_product',
    'product_id' => '0001111016338'
);

$ch = curl_init();

curl_setopt($ch, CURLOPT_URL, "https://realtime.oxylabs.io/v1/queries");
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($params));
curl_setopt($ch, CURLOPT_POST, 1);
curl_setopt($ch, CURLOPT_USERPWD, "USERNAME" . ":" . "PASSWORD");

$headers = array();
$headers[] = "Content-Type: application/json";
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);

$result = curl_exec($ch);
echo $result;

if (curl_errno($ch)) {
    echo 'Error:' . curl_error($ch);
}
curl_close($ch);

Golang

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
)

func main() {
    const Username = "USERNAME"
    const Password = "PASSWORD"

    payload := map[string]interface{}{
        "source": "citymarket_product",
        "product_id": "0001111016338",
    }

    jsonValue, _ := json.Marshal(payload)

    client := &http.Client{}
    request, _ := http.NewRequest("POST",
        "https://realtime.oxylabs.io/v1/queries",
        bytes.NewBuffer(jsonValue),
    )

    request.SetBasicAuth(Username, Password)
    response, _ := client.Do(request)

    responseText, _ := ioutil.ReadAll(response.Body)
    fmt.Println(string(responseText))
}

C#

using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Net.Http.Json;
using System.Threading.Tasks;

namespace OxyApi
{
    class Program
    {
        static async Task Main()
        {
            const string Username = "USERNAME";
            const string Password = "PASSWORD";

            var parameters = new {
                source = "citymarket_product",
                product_id = "0001111016338"
            };

            var client = new HttpClient();

            Uri baseUri = new Uri("https://realtime.oxylabs.io");
            client.BaseAddress = baseUri;

            var requestMessage = new HttpRequestMessage(HttpMethod.Post, "/v1/queries");
            requestMessage.Content = JsonContent.Create(parameters);

            var authenticationString = $"{Username}:{Password}";
            var base64EncodedAuthenticationString = Convert.ToBase64String(System.Text.ASCIIEncoding.UTF8.GetBytes(authenticationString));
            requestMessage.Headers.Add("Authorization", "Basic " + base64EncodedAuthenticationString);

            var response = await client.SendAsync(requestMessage);
            var contents = await response.Content.ReadAsStringAsync();

            Console.WriteLine(contents);
        }
    }
}

Java

package org.example;

import okhttp3.*;
import org.json.JSONObject;
import java.util.concurrent.TimeUnit;

public class Main implements Runnable {
    private static final String AUTHORIZATION_HEADER = "Authorization";
    public static final String USERNAME = "USERNAME";
    public static final String PASSWORD = "PASSWORD";

    public void run() {
        JSONObject jsonObject = new JSONObject();
        jsonObject.put("source", "citymarket_product");
        jsonObject.put("product_id", "0001111016338");

        Authenticator authenticator = (route, response) -> {
            String credential = Credentials.basic(USERNAME, PASSWORD);
            return response
                    .request()
                    .newBuilder()
                    .header(AUTHORIZATION_HEADER, credential)
                    .build();
        };

        var client = new OkHttpClient.Builder()
                .authenticator(authenticator)
                .readTimeout(180, TimeUnit.SECONDS)
                .build();

        var mediaType = MediaType.parse("application/json; charset=utf-8");
        var body = RequestBody.create(jsonObject.toString(), mediaType);
        var request = new Request.Builder()
                .url("https://realtime.oxylabs.io/v1/queries")
                .post(body)
                .build();

        try (var response = client.newCall(request).execute()) {
            if (response.body() != null) {
                try (var responseBody = response.body()) {
                    System.out.println(responseBody.string());
                }
            }
        } catch (Exception exception) {
            System.out.println("Error: " + exception.getMessage());
        }

        System.exit(0);
    }

    public static void main(String[] args) {
        new Thread(new Main()).start();
    }
}

JSON

{
    "source": "citymarket_product",
    "product_id": "0001111016338"
}

Request parameter values

Parameter	Description	Default Value
source	Sets the scraper.	citymarket_product
product_id	The 13-digit product ID, found at the end of the product page URL, e.g. citymarket.com/p/kroger-carbmaster-original-lactose-free-skim-milk/0001111016338.	-
store_id, delivery_zip, fulfillment_type	Optional localization parameters, see Localization and Filtering.	-

- mandatory parameter

Localization and Filtering

City Market shows search results and product availability for a selected store or delivery area. If none of the parameters below is set, the site falls back to a store of its own choice.

Parameter	Applies to	Description	Default
store_id	citymarket_search, citymarket_product	Selects the store. The ID is the division number and the store number from the store's page URL on citymarket.com, joined together: …/stores/grocery/co/woodland-park/gold-hill-woodland-park/620/00431 → 62000431. Required when fulfillment_type is pickup or in_store.	-
delivery_zip	citymarket_search, citymarket_product	ZIP code of the delivery area, e.g. 81601. Required when fulfillment_type is delivery.	-
fulfillment_type	citymarket_search, citymarket_product	Fulfillment method. Available values: pickup, delivery, in_store. pickup and in_store need store_id; delivery needs delivery_zip.	in_store with store_id alone, delivery with delivery_zip alone
price_range	citymarket_search	Price filter, formatted as min_price-max_price with two decimal places, e.g. 2.00-5.00.	-
brand	citymarket_search	Filters the results to one brand, e.g. Fairlife.	-

If fulfillment_type is set without the parameter it needs, the result comes back with status_code 400. A store_id of a store outside City Market, or a delivery_zip outside its delivery area, returns status_code 613.

{
    "source": "citymarket_search",
    "query": "milk",
    "store_id": "62000431",
    "fulfillment_type": "pickup",
    "brand": "Fairlife"
}