loading...

Create custom Google search engine for your domain(s) and fetch results as JSON

jochemstoel profile image Jochem Stoel ・1 min read

I decided to write this simple 5 minute tutorial before I go to sleep.

Create a Custom Google Search Engine for your website domain(s) and access it programmatically with barely any code.

Navigate to https://cse.google.com/cse/all where you can create a custom search engine. Click on the "Add" button and provide one or multiple sites to search on. In this case just add dev.to/* to the list.
If you enable the setting called 'Search the entire web', your search results will be augmented with results from the web if nothing (or not enough) is found. That means disable this to make sure it will only ever show results from your web domain(s).

s1

If you want to embed your custom search engine on a webpage with a simple copy/paste then you can choose between 7 different layouts of your search engine. I personally prefer the compact one. These layouts show advertisements.

s2

There are two options for programmatic access. The option Custom Search JSON API is free and has a limit of 10,000 queries per day. This is what you want in this case.

If your Custom Search Engine is restricted to only searching specific sites (10 or fewer), you can use the Custom Search Site Restricted JSON API. This API is similar to the JSON Custom Search API except this version has no daily query limit. Custom Search Site Restricted JSON API requests cost $5 per 1000 queries and there is no daily query limit. You may sign up for billing in the API Console.

You need

API key (get it here: https://developers.google.com/custom-search/json-api/v1/introduction)
Search Engine identifier cx (which you can find in your search engine public URL: https://cse.google.com/cse?cx=009833334622897458665:rtvizlbvdpk)

To query your search engine for 'open source', simply make a GET request to https://www.googleapis.com/customsearch/v1?key=YOUR_API_KEY&cx=SEARCH_ENGINE_ID&q=open+source

Let's say you're on top of things. Add a sort=date parameter to the querystring to sort the results by date (newest first). Google picks up on changes pretty fast as you can see in this screenshot.

s3

/* since this is a public API, it permits cross origin XMLHttpRequests from the browser */
fetch('https://www.googleapis.com/customsearch/v1?key=YOUR_API_KEY&cx=SEARCH_ENGINE_ID&q=open+source&sort=date').then(response => response.json()).then(json => {
   // json.items has the results 
}).catch(console.error)

ZzzzZZ..

Posted on Aug 9 '18 by:

jochemstoel profile

Jochem Stoel

@jochemstoel

Pellentesque nec neque ex. Aliquam at quam vitae lacus convallis pulvinar. Mauris vitae ullamcorper lacus. Cras nisi dui, faucibus non dolor quis, volutpat euismod massa. Donec et pulvinar erat.

Discussion

markdown guide
 

Hi Jochem

Thanks for this tut i have followed all the steps. I am now stock on implementing the search to my website. Can you please assist.

 

Hi Siya, you need to make a search form on your webpage. A single input that fetches the JSON from your search engine URL and then populates the body or desired HTML element with the results.

Do you need an example of how to do that?

 

Hi Jochem,

thanks a ton for this tutorial. I'm a bit of a newbie and I can't seem to find where I can put the 'a' element with href to item.link that will send it back to the

. I'm sure it's something simple but I can't seem to find it. I googled things like "add json elements in script to div" and looked at this

stackoverflow.com/questions/567779...

But no luck. Help would be appreciated thank you!

Hi Jake, there is something wrong with the formatting of your message. Did you forget a closing character?

You need to iterate the results like I showed already and create an <a> element for each result with its href attribute set to the value of item.link.

result.items.forEach(item => {
   /* create an <a> element */
   var a = document.createElement('a')
   /* set the href attribute to the value of item.link */
   a.setAttribute('href', item.link)
   /* also set a clickable text of course */
   a.innerText = item.title

   /* select the element you want to insert the link into 
      i assume here that is a div with id="results" */
   var resultsDiv = document.getElementById('results')
   /* insert the <a> into the result div */
   resultsDiv.appendChild(a)
})

Voila! And you are done.

You will notice that every link will appear on one line this way, so you want to wrap them in a <p> element or some other element that automatically does a newline. You have two options here.
Option 1 is to create an additional <p> element and inject the <a> into that, the second option is just writing the HTML you want in a single line. A little ugly but works too and nothing wrong with it. For example:

result.items.forEach(item => {
   /* basically this says, replace the innerHTML of this div 
      with what is already there + our new extra HTML */
   var results = document.getElementById('results')
   results.innerHTML = results.innerHTML + `<p><a href="${item.link}">${item.title}</a></p>`
})

If you are using jQuery it is even easier.

result.items.forEach(item => {
   $('#results').append(`<p><a href="${item.link}">${item.title}</a></p>`)
})

If you made it through the tutorial then this should really answer your question.

Ah yes I see that mistake. Thank you so much for your quick reply, this helps massively! This is exactly what I need. I tried using 'results.innerHTML' but I had passed the string incorrectly it seems. I have this working perfect now, and all of your helpful information has made it easy to understand why it works. Once again thank you so much!

 

Thank you, Jockem.
Can you please provide me with the example. I can create the search form, i am not sure what to do with the JS fetch code and how to display the results.

<!-- when the form is submitted, either with submit button or ENTER key, call search() -->
<form onsubmit="return search()">
    <input type="text" id="query">
</form>
<!-- show the results in this DIV -->
<div id="results"></div>

<script>
function search() {
    const cx = '009833334622897458665:rtvizlb' // your search engine id 
    const apikey = 'AIzaSyAmvBcHwO9VtVny-cVrP7IhXJ' // your api key

    /* select the input element */
    let input = document.getElementById('query')

    /* fetch the following URL that includes apikey, cx and the value of input */
    fetch(`https://www.googleapis.com/customsearch/v1?key=${apikey}&cx=${cx}&q=${input.value}`).then(response => response.text()).then(text => {
        let result = JSON.parse(text)
        result.items.forEach(item => {
            /* add it to your results div */
            // item.link, item.title, ...etc
        })
    })
    /* make sure the form isn't actually submitted by returning false */
    return false 
}
</script>

More details about what happens with fetch:

/* this fetches the URL and resolves response */
fetch(`https://www.googleapis.com/customsearch/v1?key=${apikey}&cx=${cx}&q=${input.value}`)

/* this resolves only the responseText */
.then(response => response.text())

/* parse the responseText to Object */
JSON.parse(text)

The Object parsed from JSON looks like this:

Each item in result.items looks like this:

I'll let it up to you figuring out how to add an item to your result DIV. (create an <a> element and set its href attribute to item.link)

Awesome, thank you so much Jochem.
I got it:)

Hi Jochem!

Great tut! I was wondering if It's possible to add the autocomplete feature on this form input

To use Google Suggest, you need a server. Cross Domain XMLHttpRequests are not allowed. I have actually created an NPM module for Google Suggest that does exactly that. It fetches suggestions from Google Search.
All you need to do is create a server that pipes the response to the client and then use some jQuery auto-complete plugin to render them visually.

Hi Jochem, thanks for the advise, very useful. I'll look into this.

Did you figure it out? Does it need an article?

Thanks for the follow up Jochem, however I'm more interested in receiving suggestions based on the sites listed in my search engine rather than the entire web. Recently I found this function (google.search.CustomSearchControl.attachAutoCompletion) so I'm going to try implement it in my project, thanks again.

 

Should probably be .catch(console.error.bind(console))

 
 

Suit yourself. What you're doing didn't make it into stable chrome until 2017.

Actually since then I thought about this dialogue with you deeply and if that notation was still unsupported so recently then you are absolutely right to point that out.

I was not aware of this, did not consider it either and I am stubborn.

;)

You should see me when I get stubborn.

 

"There are two options for programmatic access. The option Custom Search JSON API is free and has a limit of 10,000 queries per day".

As of 10-12-2019, This Google's site states "Custom Search JSON API provides 100 search queries per day for free. If you need more, you may sign up for billing in the API Console. Additional requests cost $5 per 1000 queries, up to 10k queries per day.

If you need more than 10k queries per day and your Custom Search Engine searches 10 sites or fewer, you may be interested in the Custom Search Site Restricted JSON API, which does not have a daily query limit."

 

Can u please guide me a steps how can i implement this in wordpress?

 

No I can not. Implementing it in Wordpress is the same as what you see here only you have to copy paste into your template or a widget. You can do this yourself.

 

im confused which template you are referring to i have a JSON api key and the cx id and <async code like this

can u show me the final template that i have to copy paste in searchform.php?

 

Hey Jochem,

I'm a total coding nitwit, and I'm a (bit) embarrassed to ask you this, but:
As I'd like to use search more then 10 sites with my GCSE, I want to create a webpage using the API, with search form and the normallooking google-results (as you have made). But really, I can't get it to work.
I'v searched for examples of this, and yours is the most specific I've found so far.
However, I just can't get it to work.
Could you show an example of the code needed to implement this in my googlesites webpage (html, js and css), as you have made for your 'opensource-example'?

Thx a lot!

 

Hey. Follow me here on Dev.to, I will follow you too. Then we can talk through PM and I will gladly help you through the whole process. This keeps this page on point/topic and not side tracked. :)