Table Of Content

How To Scrape Google Trends Without Using API?

how-to-scrape-google-trends-data-without-an-api

Publish Date

July 16, 2021

Author

Scraping Intelligence

Let us execute some code as we already have Python & an IDE/Selenium installed. Now, setup a web crawler and move ahead.

#IMPORT THESE PACKAGES
import selenium
from selenium import webdriver
#OPTIONAL PACKAGE, BUY MAYBE NEEDED
from webdriver_manager.chrome import ChromeDriverManager

Let’s use the following line of code to define our web browser in Selenium:

#THIS INITIALIZES THE DRIVER (AKA THE WEB BROWSER)
                    driver = webdriver.Chrome(ChromeDriverManager().install())

We’ll use the search phrase “cars” to acquire the URL of a specific Google Trends search page. Let’s head over to scraping of Google Trends and look for automobiles:

As you’ll see, there is a lot of information on this page. Let’s imagine we wanted to use scraped Google Trends to get the most common relevant search term:

Then, in your Python script, copy and paste the following code:

POSTS = driver.find_element_by_xpath(‘DELETE THESE WORDS AND REPLACE WITH XPATH’).text

Return to the Google Chrome browser now and save the XPath. To do so, right-click on the line in which the XPath appeared in the inspector tab, then right-click > copy > copy complete XPath.

Return to your Python script and insert the following between the two quotes:

POSTS = driver.find_element_by_xpath(‘/html/body/div[2]/div[2]/div/md-content/div/div/div[4]/trends-widget/ng-include/widget/div/div/ng-include/div/div[1]/div/ng-include/a/div/div[2]/span’).text
                    print(POSTS)

That concludes our discussion. In our console, we’ll now be able to see the top related query for that search term:

To accomplish this, return to your Python script and cut/paste the following code:

#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO
                    driver.get('COPY AND PASTE YOUR URL HERE')

This code should be replaced with the following URL from our Google Trends search page:

#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO
                    driver.get('https://trends.google.com/trends/explore?q=cars&geo=US')

Next, we’ll use the web browser to get the HTML property of the topmost related query. To begin, learn how and when to enable your development settings in your internet browser and turn them on. Next, right-click on the real top search engine (as shown below) and select inspect; you should see something similar to this:

Here is the final execution of the project

#IMPORT THESE PACKAGES
import selenium
from selenium import webdriver
#OPTIONAL PACKAGE, BUY MAYBE NEEDED
from webdriver_manager.chrome import ChromeDriverManager

#THIS INITIALIZES THE DRIVER (AKA THE WEB BROWSER)
driver = webdriver.Chrome(ChromeDriverManager().install())

#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO
driver.get('https://trends.google.com/trends/explore?q=cars&geo=US')

POSTS = driver.find_element_by_xpath('/html/body/div[2]/div[2]/div/md-content/div/div/div[4]/trends-widget/ng-include/widget/div/div/ng-include/div/div[1]/div/ng-include/a/div/div[2]/span').text
print(POSTS)

Once the process gets completed, you have finally scraped some information from Google Trends without using an API. The above mentioned is a script used for scraping Google trends.

About the author

Zoltan Bettenbuk

Zoltan Bettenbuk is the CTO of ScraperAPI - helping thousands of companies get access to the data they need. He’s a well-known expert in data processing and web scraping. With more than 15 years of experience in software development, product management, and leadership, Zoltan frequently publishes his insights on our blog as well as on Twitter and LinkedIn.