Create A Google Trends Keyword Scraper In Python

Spread the love

For my Bing Rewards Auto Searcher python script I was using a word list and randomly combining words to create phrases. This worked fine but I worried that the searches would be flagged by Bing because, well, they made no sense. So I looked for a way to easily gather a list of legitimate search phrases and there is no better place to get that than Google Trends. Google is nice and provides an RSS feed of their daily search trends so I went about creating a Google Trends keyword scraper that would parse that list and create a simple text file of each keyword that ends up trending. Below is the script.

All of my code is run on Python 3.7. If you need help on installing Python you can check out my Windows 10 and Centos 7.5 installation guides.

Google Trends Keyword Scraper

The below script is pretty simple. Just copy and run, making sure that you create all the needed files and adjust the locations used below. You can do many things with this but I simply need it to create a txt file with a rolling list of the keywords.

import codecs
import os
import urllib.request
from xml.etree import ElementTree

url = 'https://trends.google.com/trends/trendingsearches/daily/rss?geo=FR'
keywords_filename = './keywords.txt'
tmp_filename = './keywords_tmp.xml'
urllib.request.urlretrieve(url, tmp_filename)
if os.path.isfile(keywords_filename):
    with open(keywords_filename, 'r') as f:
        old_list = f.read().splitlines()
else:
    old_list = []
tree = ElementTree.parse(tmp_filename)
root = tree.getroot()
fresh_list = [title.text.lower()
              for title in root.iter('title')]
try:
    fresh_list.remove('daily search trends')
except ValueError:
    pass
output = '\n'.join(a for a in list(set(old_list + fresh_list)))
with codecs.open('./keywords.txt', 'w+', "utf-8") as f:
    f.write(output)
os.remove(tmp_filename)

Live Example

I have made my list public so that anyone that wishes to use it can. You can check it out at https://www.myhelpfulguides.com/keywords.txt. The script that powers my list runs every 5 minutes so it should be very up to date.


Spread the love

8
Leave a Reply

avatar
3 Comment threads
5 Thread replies
3 Followers
 
Most reacted comment
Hottest comment thread
3 Comment authors
Olivier PonsBrent WilsonAmin Recent comment authors
  Subscribe  
newest oldest most voted
Notify of
Amin
Guest
Amin

This is very nice application. I am actually going to bookmark this.

One note. The strings in Google trend you are considering are not completely upper/lower. For example I saw amber alert and AMBER alert in the list. Having that considered would make it a more realistic frequency counter.

Olivier Pons
Guest

And instead of your very long function which suggests you’re coming from Php world or Java world where you have to code a lot of things by hand, you just can cast into a set which a “list” of unique elements: newList = list(set(oldList + freshList)) and avoid “newList” use python-style variables like “new_list”. You can also avoid hard-coded names and use current path (just use “./filename”) much more portable. And you can avoid things like this: oldList = [] for data in old: oldList.append(data) just make a deepcopy: old_list = deepcopy(data) All in all here’s your code, half the… Read more »

Olivier Pons
Guest

f.close is nonsense, it’s f.close()