Google trends will provide you with super powers of analyzing data. Check the script to scrape Google Trends without using API.
Let us execute some code as we already have Python & an IDE/Selenium installed. Now, setup a web crawler and move ahead.
#IMPORT THESE PACKAGES import selenium from selenium import webdriver #OPTIONAL PACKAGE, BUY MAYBE NEEDED from webdriver_manager.chrome import ChromeDriverManager
Let’s use the following line of code to define our web browser in Selenium:
#THIS INITIALIZES THE DRIVER (AKA THE WEB BROWSER) driver = webdriver.Chrome(ChromeDriverManager().install())
We’ll use the search phrase “cars” to acquire the URL of a specific Google Trends search page. Let’s head over to scraping of Google Trends and look for automobiles:
As you’ll see, there is a lot of information on this page. Let’s imagine we wanted to use scraped Google Trends to get the most common relevant search term:
Then, in your Python script, copy and paste the following code:
POSTS = driver.find_element_by_xpath(‘DELETE THESE WORDS AND REPLACE WITH XPATH’).text
Return to the Google Chrome browser now and save the XPath. To do so, right-click on the line in which the XPath appeared in the inspector tab, then right-click > copy > copy complete XPath.
Return to your Python script and insert the following between the two quotes:
POSTS = driver.find_element_by_xpath(‘/html/body/div[2]/div[2]/div/md-content/div/div/div[4]/trends-widget/ng-include/widget/div/div/ng-include/div/div[1]/div/ng-include/a/div/div[2]/span’).text print(POSTS)
That concludes our discussion. In our console, we’ll now be able to see the top related query for that search term:
To accomplish this, return to your Python script and cut/paste the following code:
#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO driver.get('COPY AND PASTE YOUR URL HERE')
This code should be replaced with the following URL from our Google Trends search page:
#THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO driver.get('https://trends.google.com/trends/explore?q=cars&geo=US')
Next, we’ll use the web browser to get the HTML property of the topmost related query. To begin, learn how and when to enable your development settings in your internet browser and turn them on. Next, right-click on the real top search engine (as shown below) and select inspect; you should see something similar to this:
Here is the final execution of the project
#IMPORT THESE PACKAGES import selenium from selenium import webdriver #OPTIONAL PACKAGE, BUY MAYBE NEEDED from webdriver_manager.chrome import ChromeDriverManager #THIS INITIALIZES THE DRIVER (AKA THE WEB BROWSER) driver = webdriver.Chrome(ChromeDriverManager().install()) #THIS PRETTY MUCH TELLS THE WEB BROWSER WHICH WEBSITE TO GO TO driver.get('https://trends.google.com/trends/explore?q=cars&geo=US') POSTS = driver.find_element_by_xpath('/html/body/div[2]/div[2]/div/md-content/div/div/div[4]/trends-widget/ng-include/widget/div/div/ng-include/div/div[1]/div/ng-include/a/div/div[2]/span').text print(POSTS)
Once the process gets completed, you have finally scraped some information from Google Trends without using an API. The above mentioned is a script used for scraping Google trends.
Zoltan Bettenbuk is the CTO of ScraperAPI - helping thousands of companies get access to the data they need. He’s a well-known expert in data processing and web scraping. With more than 15 years of experience in software development, product management, and leadership, Zoltan frequently publishes his insights on our blog as well as on Twitter and LinkedIn.
Explore our latest content pieces for every industry and audience seeking information about data scraping and advanced tools.
No matter what industry you belong to, web scraping helps extract insights from industry datasets. It is a systematic process of getting data from online sources, top-ranking websites, popular platforms, and databases.
Learn how to scrape alcohol pricing & market trends safely. Explore legal risks, best tools, and strategies for extracting beverage industry data efficiently.
Learn how to collect real-time data from Google Shopping, which has an array of products and simple steps to scrape price and product data from Google Shopping.