How to Scrape Google Shopping for Price and Product Data?

In today's highly competitive business arena, it is difficult to picture a scenario where retailers or e-commerce companies expect relevancy without leveraging web scraping. E-commerce is always going to be competitive. The fact is, that many retailers likely have different marketing strategies on various platforms for a single item. When you compare product information across different platforms scraping Google Shopping will save you time.

Google Shopping, previously known as Product Listing Ads, is an online shopping service that enables users to search for and compare products from various e-commerce websites. This platform allows you to quickly and easily compare prices and product details across multiple providers. In this guide, you will learn what Google Shopping is and how to extract data from it. Additionally, we will briefly address the legality of scraping Google Shopping and the challenges you may encounter during the process.

What is Google Shopping?

Google Shopping is a Google service that lets people search for and compare products from various online retailers. Previously known as Google Products Search, Froogle, and Google Products. With Google Shopping, users choose their products, as it provides product listings showing prices, images, seller names, and reviews; listings showing the information can appear in the Shopping tab, or ads can appear in the normal Google search results.

Google Shopping allows consumers to find the best deal on thousands of brands and allows retailers to make their products available for users. When a user clicks a product link, they are taken to the vendor's page to purchase, thus Google shopping serves as a vehicle for companies to submit their products online.

What You Can Scrape from Google Shopping?

Scraping data from Google Shopping can be useful in different aspects with a large amount of data representing competitive advantages and opportunities for expansion in your business. Google Shopping allows you to extract useful information, including product titles, descriptions, prices, discounts, and availability. You can also identify product ratings, customer reviews, retailer names, product categories, tags, and images. Accessing this data means you can get insight into your competition, analyze their pricing, identify trends in customer feedback, and improve your product listings. Businesses can make better-informed decisions, improve their SEO, and increase sales by improving product imagery and competitive pricing in an online channel.

Benefits of scraping Google Shopping

Google Shopping is extremely beneficial for e-commerce stores. The main benefit is it is time-saving. Instead of searching and checking different websites, you can get all the pricing and product details from a single source. By getting information like this, you will have tremendous benefits for your business in these areas:

Price Monitoring

Google Shopping allows easy pricing comparisons. Regularly scraping prices for your products will let you know where your competitors are pricing their products giving you the ability to make a smart pricing strategy.

Competitor Analysis

Scraping Google Shopping lets you extract detailed product data like sizes, colors, materials, and brands. This data will help you learn the competitor's strengths, and weaknesses, and how you best position your products in the market.

Competitive Pricing Analysis

Price is often the determining factor for a customer’s buying decision. Scraping Google Shopping provides you with the ability to monitor competitor pricing in real-time, thus helping you price your products competitively, as well as optimizing sales.

Product Availability Monitoring

When scraping you will gather data about products that are in-stock, out-of-stock, or on-sale. Product availability monitoring has the clear benefit of making better inventory planning from a customer demand standpoint.

Trend Analysis and Market Insight

Scraping can give you general insights about product trends, and products that customers are trending towards. This can provide useful information for product development, marketing campaigns, and keeping up with a rapidly changing marketplace.

Improved Product Listing Information

You can identify popular images, product descriptions, and keywords by analyzing your competitor's product listings. It will help you improve each of your product listings through the A-B-C process and enhance your SEO, which can drive more traffic and ultimately increase sales and conversions.

Is it Legal to Scrape Google Shopping Data?

It is generally legal to scrape Google Shopping because you are scraping public information and not private or restricted information. Everyone should know that scraping and hacking are not the same. Legality only matters in regards to how the scraping is performed and the use of any data.

Many times, it is legal to scrape public information, but it may violate a platform's Terms of Service (ToS). We strongly encourage everybody to read through Google ToS, at a minimum, before starting any scraper. Scraping responsibly while adhering to the structure of the site, rate limits, and legal use, is very important to avoid blocking and possibly worse outcomes.

Does Google Shopping offer any APIs?

In the past, Google offered a Shopping Search API, but it has been no longer available since September 16, 2013. For many retailers, the implicit end to Google Shopping APIs made it more difficult to have overall market views using Google vendors.

Today, Google has a Content API for Shopping designed explicitly for merchants only (not scraping or market research purposes). It allows sellers to upload and manage their product listings or advertisements in real-time. Unfortunately, the Content API for Shopping does not allow access to listings or prices from competitors.

In short, if you want to extract market data from Google Shopping for analysis or comparison, you cannot do it or receive a guarantee by using any of Google's official APIs—you will have to create custom web scraping tools or work with scraping services like Scraping Intelligence.

Understanding Google Shopping Data Architecture

Google Shopping operates like a search engine for consumers to find, compare, and purchase products corresponding to price and presenting thousands of vendors all in the same spot.

Google Shopping integrates several features that help users encounter the right option, including the ability to sort products by price, brand, shipping options, or sale status.

Once you click on a product, Google Shopping provides additional context for the product and redirects you to the seller’s product page for purchase.

It should be noted that Google Shopping operates more like a directory and provides no other e-commerce directly. This allows Google Shopping to be an ideal source of product data. You can scrape thousands of product listings ranking for your targeted keywords.

The Setbacks of Scraping Google Shopping

Scraping Google Shopping is not really easy. Google Shopping has a lot of anti-scraping features built in to block bots from scraping and collecting large-scale data. If you do not have the proper tools or setup, the chances are your scraper will just get blocked immediately. Here are the most common setbacks of scraping:

Rate Limiting

Google limits the number of requests a single IP address can make. Submitting requests too fast can result in blocks.

Browser Fingerprinting

Google tracks attributes of your browser and screen resolution etc. to determine bot behavior.

CAPTCHAs

Google uses CAPTCHA that often cannot be "solved" by simple scrapers in order to ensure the request is coming from a human.

JavaScript Rendering

Google Shopping leverages Javascript to load the content. Most scrapers are able to scrape but cannot see the dynamic content unless it is running headless browsers to fully render pages.

How to Avoid These Issues

If you want to avoid this hassle, use automated tools like Scraping Intelligence's Google Shopping Endpoint and be able to scrape Google Shopping data as JSON objects with a single API call, saving you time, effort, and money.

Step-by-Step Guide for Scraping Google Shopping Using Our API

availability. In this tutorial, we will demonstrate how to scrape Google Shopping search, product, and pricing data using Scraping Intelligence API and Python.

Step 1: Install Python & Required Libraries

Make sure you have Python 3.6+ installed. Next, you will want to install the necessary Python packages:

pip install requests pandas

requests: To send the HTTP request

pandas: Organizing and exporting data

Step 2: Create a Payload for the Google Shopping Search Page

At this point, you will create the payload that sets up your API request.

Example Search Term: “levis”

payload = {
   'source': 'google_shopping_search',
   'domain': 'com',
   'query': 'levis',
   'pages': 1,
   'context': [
       {'key': 'sort_by', 'value': 'pd'},
       {'key': 'min_price', 'value': 30},
   ],
   'parse': 'true',
}

Step 3: Make the POST Request with Authentication

You will now make the request using your Scraper API endpoint and authentication details.

response = requests.request(
   'POST',
   'https://realtime.oxylabs.io/v1/queries',
   auth=('username', 'password'),
   json=payload,
)

Step 4: Get the Data From the JSON Response

Next, we will extract the data such as Product Title, Price, and Store Name.

#Get the content from the response
result=response.json()['results'][0]['content']
products = result['results']['organic']


#Create a DataFrame
df = pd.DataFrame(columns=['Product Title', 'Price', 'Store'])


#iterate through all the products
for p in products:
   title = p['title']
   price = p['price_str']
   store = p['merchant']['name']
   df = pd.concat([pd.DataFrame([[title, price, store]], columns=df.columns),
                   df], ignore_index=True)

Step 5: Saving to CSV and JSON file

df.to_csv('google_shopping_search.csv', index=False)
df.to_json('google_shopping_search.json', orient='split', index=False)

Full Script for Search Page

import pandas as pd
import requests


# Structure payload
payload = {
   'source': 'google_shopping_search',
   'domain': 'com',
   'query': 'levis',
   'pages': 1,
   'context': [
       {'key': 'sort_by', 'value': 'pd'},
       {'key': 'min_price', 'value': 30},
   ],
   'parse': 'true',
}


# Get response
response = requests.request(
   'POST',
   'https://realtime.oxylabs.io/v1/queries',
   auth=('username', 'password'),
   json=payload,


)


#Get the content from the response
result=response.json()['results'][0]['content']
products = result['results']['organic']


#Create a DataFrame
df = pd.DataFrame(columns=['Product Title', 'Price', 'Store'])


#iterate through all the products
for p in products:
   title = p['title']
   price = p['price_str']
   store = p['merchant']['name']
   df = pd.concat([pd.DataFrame([[title, price, store]], columns=df.columns),
                   df], ignore_index=True)


#Copy the DataFrame to CSV and JSON files
df.to_csv('google_shopping_search.csv', index=False)
df.to_json('google_shopping_search.json', orient='split', index=False)

Final Thoughts

Scraping Google Shopping is essential, if you want accurate information regarding the products and prices of your major competitors and ultimately make data-driven decisions to grow your business. If you are looking to elevate your scraping efforts, you can purchase proxies, like residential proxies, in order to yield a smoother and more efficient scraping experience. We hope this guide was understandable to you and will help you in making data collection easier for your needs. If you still have any questions about our Google Scraper API, contact Scraping Intelligences’ professional team will be more than happy to help you.

About the author

---

Zoltan Bettenbuk

Zoltan Bettenbuk is the CTO of ScraperAPI - helping thousands of companies get access to the data they need. He’s a well-known expert in data processing and web scraping. With more than 15 years of experience in software development, product management, and leadership, Zoltan frequently publishes his insights on our blog as well as on Twitter and LinkedIn.