I reverse engineered 2 Big Keyword Research Tools, and I found out how they work and how you can build the same on your own.
The Sites Are:
Answerthepublic: a website that allows users to type in a keyword or phrase and receive a list of related questions that people have searched for on the internet.
Keywordtool.io: a keyword research and analysis tool that helps users find and analyze the best keywords for their website or online business.
Even if you are not so experienced in programming, you can still easily follow up with this tutorial and Develop a Keyword Research Tool with Python or C#.
How do Keyword Research Tools Work?
Well, it depends on the scope and functionality of the Tools you’re making, and the SEO metrics you want your Tool to Provide.
In our case, we will focus on “google search suggestions” and “Question Suggestions.” I will also tell you later how you can get more metrics.
Let’s see how these tools are getting results, Open Google Search, and Type “Python Tutorial” press space and type the letter “f”, you will see Google suggesting keyword ideas and search queries!
And if you Add “how” before the “python tutorial” keyword in Google Search. you will get question suggestions.
That's it!
How To Read Google autosuggestions programmatically?
We have two approaches:
- Using Web Automation and Scraping
- Using Google Search Free API Call
I chose the second approach simply because with web automation, the operation will be slower, and you may get Re-captcha to solve, and things will be complicated.
API Call for Google Keyword Suggestions:
open your browser, and paste this url:
http://google.com/complete/search?output=toolbar&gl=COUNTRY&q=Your_QUERY
You can change the country and the query to get google autosuggestions in XML format using this call as shown below:
I think you got the point, we will call the Google Suggestions API, then fetch the results from our code.
Python code:
import requests
import xml.etree.ElementTree as ET
# Define Class
class QuestionsExplorer:
def GetQuestions(self, questionType, userInput, countryCode):
questionResults = []
# Build Google Search Query
searchQuery = questionType + " " + userInput + " "
# API Call
googleSearchUrl = "http://google.com/complete/search?output=toolbar&gl=" + \
countryCode + "&q=" + searchQuery
# Call The URL and Read Data
result = requests.get(googleSearchUrl)
tree = ET.ElementTree(ET.fromstring(result.content))
root = tree.getroot()
for suggestion in root.findall('CompleteSuggestion'):
question = suggestion.find('suggestion').attrib.get('data')
questionResults.append(question)
return questionResults
# Get a Keyword From The User
userInput = input("Enter a Keyword: ")
# Create Object of the QuestionsExplorer Class
qObj = QuestionsExplorer()
# Call The Method and pass the parameters
questions = qObj.GetQuestions("is", userInput, "us")
# Loop over the list and pring the questions
for result in questions:
print(result)
print("Done")
C# code:
//start
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Xml.Linq;
namespace GoogleSuggestion
{
//Define Class
class QuestionsExplorer
{
public List<string> GetQuestions(string questionType, string userInput, string countryCode)
{
var questionResults = new List<string>(); //List To Return
//Build Google Search Query
string searchQuery = questionType + " " + userInput + " ";
string googleSearchUrl =
"http://google.com/complete/search?output=toolbar&gl=" + countryCode + "&q=" + searchQuery;
//Call The URL and Read Data
using (HttpClient client = new HttpClient())
{
string result = client.GetStringAsync(googleSearchUrl).Result;
//Parse The XML Documents
XDocument doc = XDocument.Parse(result);
foreach (XElement element in doc.Descendants("CompleteSuggestion"))
{
string question = element.Element("suggestion")?.Attribute("data")?.Value;
questionResults.Add(question);
}
}
return questionResults;
}
}
class Program
{
static void Main(string[] args)
{
//Get a Keyword From The User
Console.WriteLine("Enter a Keyword:");
var userInput = Console.ReadLine();
//Create Object of the QuestionsExplorer Class
var qObj = new QuestionsExplorer();
//Call The Method and pass the parameters
var questions = qObj.GetQuestions("what", userInput, "us");
//Loop over the list and pring the questions
foreach (var result in questions)
{
Console.WriteLine(result);
}
//Finish
Console.WriteLine();
Console.WriteLine("---Done---");
Console.ReadLine();
}
}
}
//end
Get Source Codes on Github:
Conclusion
In this post, I showed you how you can build a keyword research tool based on Google's Suggestion API with the power of two popular programming languages, C# and Python.
Maybe you are wondering how to get search metrics, like CPC, Monthly Search Volume, and Competition. The answer is in this post.
Take care, and Happy Coding!
Top comments (2)
Hi, thanks for the code but the answer suppose to be in the native country code and not always english. That means search for keyword in france has to come with french keywords mostly and the google should be google.fr.
Nice content Hasan. You have a fan!