Table Of Content

How To Scrape Google Trends Without Using API?

how-to-scrape-google-trends-data-without-an-api

Publish Date

July 16, 2021

Author

Scraping Intelligence

Let us execute some code as we already have Python & an IDE/Selenium installed. Now, setup a web crawler and move ahead.

#IMPORT THESE PACKAGES
import selenium
from selenium import webdriver
#OPTIONAL PACKAGE, BUY MAYBE NEEDED
from webdriver_manager.chrome import ChromeDriverManager

Let’s use the following line of code to define our web browser in Selenium:

#THIS INITIALIZES THE DRIVER (AKA THE WEB BROWSER)
                    driver = webdriver.Chrome(ChromeDriverManager().install())

We’ll use the search phrase “cars” to acquire the URL of a specific Google Trends search page. Let’s head over to scraping of Google Trends and look for automobiles:

As you’ll see, there is a lot of information on this page. Let’s imagine we wanted to use scraped Google Trends to get the most common relevant search term:

Then, in your Python script, copy and paste the following code:

POSTS = driver.find_element_by_xpath(‘DELETE THESE WORDS AND REPLACE WITH XPATH’).text

Return to the Google Chrome browser now and save the XPath. To do so, right-click on the line in which the XPath appeared in the inspector tab, then right-click > copy > copy complete XPath.

Return to your Python script and insert the following between the two quotes:

POSTS = driver.find_element_by_xpath(‘/html/body/div[2]/div[2]/div/md-content/div/div/div[4]/trends-widget/ng-include/widget/div/div/ng-include/div/div[1]/div/ng-include/a/div/div[2]/span’).text
                    print(POSTS)

That concludes our discussion. In our console, we’ll now be able to see the top related query for that search term:

To accomplish this, return to your Python script and cut/paste the following code:

#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO
                    driver.get('COPY AND PASTE YOUR URL HERE')

This code should be replaced with the following URL from our Google Trends search page:

#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO
                    driver.get('https://trends.google.com/trends/explore?q=cars&geo=US')

Next, we’ll use the web browser to get the HTML property of the topmost related query. To begin, learn how and when to enable your development settings in your internet browser and turn them on. Next, right-click on the real top search engine (as shown below) and select inspect; you should see something similar to this:

Here is the final execution of the project

#IMPORT THESE PACKAGES
import selenium
from selenium import webdriver
#OPTIONAL PACKAGE, BUY MAYBE NEEDED
from webdriver_manager.chrome import ChromeDriverManager

#THIS INITIALIZES THE DRIVER (AKA THE WEB BROWSER)
driver = webdriver.Chrome(ChromeDriverManager().install())

#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO
driver.get('https://trends.google.com/trends/explore?q=cars&geo=US')

POSTS = driver.find_element_by_xpath('/html/body/div[2]/div[2]/div/md-content/div/div/div[4]/trends-widget/ng-include/widget/div/div/ng-include/div/div[1]/div/ng-include/a/div/div[2]/span').text
print(POSTS)

Once the process gets completed, you have finally scraped some information from Google Trends without using an API. The above mentioned is a script used for scraping Google trends.

About the Author

Scraping Intelligence

Scraping Intelligence Editorial Team is a collective of data specialists, analysts, and researchers with expertise in web scraping, data extraction, and market intelligence. The team produces well-researched guides, actionable insights, and industry-focused resources that help businesses unlock the value of data and make informed, strategic decisions.