How to scrape google search results using Python?

Why Python for scraping google search results?

Considering the ease of the language and its available community on the internet to resolve code-related problems while developing any code, Python is one of the most preferred languages for web data scraping.

Further, many answers are already available on the forums like GitHub, StackOverflow, Quora, etc. These features related to python help to use it to scrape search results without any hassle.

Let's Understand Scraping Google Search Results with Python

Here, we'll understand the process to scrape Google search results with the basic python script to collect the data for the first page of Google search results.

Using this python script, you'll get a JSON file that will consist of a title, description, position, and the URL of the first 10 ranking pages on Google. You can mold this data as per your requirements and use it for product verification, market research, SEO, etc.

Basic requirements to scrape Google search results

Usually, there are parts in Google scraping using Python.

• Data collection with the HTTP request
• Extracting important data from the collected file by parsing the HTML DOM.

Python tools used for Google search scraping

• Beautiful Soup This python library helps to extract the data from HTML and XML files.
• RequestsIt is a handy tool for you to send HTTP requests.
• Residential ProxyThis proxy tool helps you to grab HTML code of the targeted URL

Setup

We know tools to collect Google search data. Now, let's start taking action to grab the data.

It's a pretty simple setup. You just need to create a new folder for this step and install beautiful soup, and request tools using below-mentioned commands in Python.

mkdir scraper pip install beautifulsoup4 pip install requests

Now, create a new file in the created folder with any name. For example, googlesearch.py

Import the installed libraries in this file.

from bs4 import BeautifulSoup import requests

The Process

Since the setup is done, you can go for the main process to extract the Google search results. Send a GET request to the target link and get the raw HTML data. Once it is done, you must go for scraping Google search data for the first 10 results as per SERP. Here is the code for this scraping.

The above code will give you an HTML code of the targeted URL. Now you'll have to use the Beautiful Soup to parse that HTML code. As given below

soup = BeautifulSoup(html.text, 'html.parser')

Since google doesn't like this process, it'll keep making certain changes to the class g from time to time. You can track that when you inspect the google page.

In this case, you'll grab all the classes with the name g

allData = soup.find_all("div",{"class":"g"})

After getting the data for class g, you'll be running a loop and reach out to each and every item in the alldata list.

To find the URL of the website, title and the description, you'll have to use for loop in and check a tag, h3 tag and a span tag respectively with class aCOpRe .

In this method, Google doesn't give you 100 percent legit data, you'll have to filter it out with find() method.

Further, on printing the list data, you'll get the expected output as shown below.

Please note that Google will not allow you to scrape the data from search results after a certain number of requests.

Process to Scraping Google Search Results from Different Countries using Python Language

Since we've discussed web data scraping in the above part of the blog, let's go to some advanced tactics to extract the data from Google searches in different countries.

Since you'll be scraping the data from different country origins, you'll have to use a residential proxy to get the output.

There are plenty of web data Scraping tools available, you can use any of your choice as a google search result scraper.

You just have to enter the URL you wish to grab the data for in the particular data scraping tool, and it'll do the rest.

Here we're using a proxy server with a list of 10 user agents. Where you can use them and rotate the proxies using a residential proxy provider so that will change the origin of request every time.

There are different codes for different country origins like us for USA, ca for Canada, gb for England, etc.

The code to extract the data will be same as that used before.

In this method, Google won't be able to blog you since you're not using proxy and random user agent for every request.

You'll get the below mentioned output if you have chosen US.

And for UK, here is the output given below.

Similar to these, you can get the different outputs for different countries.

However, if you are not getting how to get the data, we are here to help you scrape the Google search results as per your expectations at Iweb Data Scraping. What are the Limitations of scraping Googling search results with Python? Though Python is one of the leading coding languages for scraping Google search results, there are still some drawbacks involved. It is because Python is a dynamic language. Further it gives many runtime errors, and can't cope up with the multiple threads like other languages.

Conclusion

We hope this was helpful for you to know about how to scrape Google search results using Python. If you're looking for data from Google search but unable to scrape on your own, Iweb Data Scraping is happy to help you any time.