DEV Community

loading...

Create custom Google search engine for your domain(s) and fetch results as JSON

jochemstoel profile image Jochem Stoel ・2 min read

I decided to write this simple 5 minute tutorial before I go to sleep.

Create a Custom Google Search Engine for your website domain(s) and access it programmatically with barely any code.

Navigate to https://cse.google.com/cse/all where you can create a custom search engine. Click on the "Add" button and provide one or multiple sites to search on. In this case just add dev.to/* to the list.
If you enable the setting called 'Search the entire web', your search results will be augmented with results from the web if nothing (or not enough) is found. That means disable this to make sure it will only ever show results from your web domain(s).

s1

If you want to embed your custom search engine on a webpage with a simple copy/paste then you can choose between 7 different layouts of your search engine. I personally prefer the compact one. These layouts show advertisements.

s2

There are two options for programmatic access. The option Custom Search JSON API is free and has a limit of 10,000 queries per day. This is what you want in this case.

If your Custom Search Engine is restricted to only searching specific sites (10 or fewer), you can use the Custom Search Site Restricted JSON API. This API is similar to the JSON Custom Search API except this version has no daily query limit. Custom Search Site Restricted JSON API requests cost $5 per 1000 queries and there is no daily query limit. You may sign up for billing in the API Console.

You need

API key (get it here: https://developers.google.com/custom-search/json-api/v1/introduction)
Search Engine identifier cx (which you can find in your search engine public URL: https://cse.google.com/cse?cx=009833334622897458665:rtvizlbvdpk)

To query your search engine for 'open source', simply make a GET request to https://www.googleapis.com/customsearch/v1?key=YOUR_API_KEY&cx=SEARCH_ENGINE_ID&q=open+source

Let's say you're on top of things. Add a sort=date parameter to the querystring to sort the results by date (newest first). Google picks up on changes pretty fast as you can see in this screenshot.

s3

/* since this is a public API, it permits cross origin XMLHttpRequests from the browser */
fetch('https://www.googleapis.com/customsearch/v1?key=YOUR_API_KEY&cx=SEARCH_ENGINE_ID&q=open+source&sort=date').then(response => response.json()).then(json => {
   // json.items has the results 
}).catch(console.error)

ZzzzZZ..

Discussion (27)

pic
Editor guide
Collapse
makwelos profile image
Siya

Hi Jochem

Thanks for this tut i have followed all the steps. I am now stock on implementing the search to my website. Can you please assist.

Collapse
jochemstoel profile image
Jochem Stoel Author

Hi Siya, you need to make a search form on your webpage. A single input that fetches the JSON from your search engine URL and then populates the body or desired HTML element with the results.

Do you need an example of how to do that?

Collapse
shnackob profile image
Jake / Smack

Hi Jochem,

thanks a ton for this tutorial. I'm a bit of a newbie and I can't seem to find where I can put the 'a' element with href to item.link that will send it back to the

. I'm sure it's something simple but I can't seem to find it. I googled things like "add json elements in script to div" and looked at this

stackoverflow.com/questions/567779...

But no luck. Help would be appreciated thank you!

Thread Thread
jochemstoel profile image
Jochem Stoel Author

Hi Jake, there is something wrong with the formatting of your message. Did you forget a closing character?

You need to iterate the results like I showed already and create an <a> element for each result with its href attribute set to the value of item.link.

result.items.forEach(item => {
   /* create an <a> element */
   var a = document.createElement('a')
   /* set the href attribute to the value of item.link */
   a.setAttribute('href', item.link)
   /* also set a clickable text of course */
   a.innerText = item.title

   /* select the element you want to insert the link into 
      i assume here that is a div with id="results" */
   var resultsDiv = document.getElementById('results')
   /* insert the <a> into the result div */
   resultsDiv.appendChild(a)
})

Voila! And you are done.

You will notice that every link will appear on one line this way, so you want to wrap them in a <p> element or some other element that automatically does a newline. You have two options here.
Option 1 is to create an additional <p> element and inject the <a> into that, the second option is just writing the HTML you want in a single line. A little ugly but works too and nothing wrong with it. For example:

result.items.forEach(item => {
   /* basically this says, replace the innerHTML of this div 
      with what is already there + our new extra HTML */
   var results = document.getElementById('results')
   results.innerHTML = results.innerHTML + `<p><a href="${item.link}">${item.title}</a></p>`
})

If you are using jQuery it is even easier.

result.items.forEach(item => {
   $('#results').append(`<p><a href="${item.link}">${item.title}</a></p>`)
})

If you made it through the tutorial then this should really answer your question.

Thread Thread
shnackob profile image
Jake / Smack

Ah yes I see that mistake. Thank you so much for your quick reply, this helps massively! This is exactly what I need. I tried using 'results.innerHTML' but I had passed the string incorrectly it seems. I have this working perfect now, and all of your helpful information has made it easy to understand why it works. Once again thank you so much!

Collapse
makwelos profile image
Siya

Thank you, Jockem.
Can you please provide me with the example. I can create the search form, i am not sure what to do with the JS fetch code and how to display the results.

Thread Thread
jochemstoel profile image
Jochem Stoel Author
<!-- when the form is submitted, either with submit button or ENTER key, call search() -->
<form onsubmit="return search()">
    <input type="text" id="query">
</form>
<!-- show the results in this DIV -->
<div id="results"></div>

<script>
function search() {
    const cx = '009833334622897458665:rtvizlb' // your search engine id 
    const apikey = 'AIzaSyAmvBcHwO9VtVny-cVrP7IhXJ' // your api key

    /* select the input element */
    let input = document.getElementById('query')

    /* fetch the following URL that includes apikey, cx and the value of input */
    fetch(`https://www.googleapis.com/customsearch/v1?key=${apikey}&cx=${cx}&q=${input.value}`).then(response => response.text()).then(text => {
        let result = JSON.parse(text)
        result.items.forEach(item => {
            /* add it to your results div */
            // item.link, item.title, ...etc
        })
    })
    /* make sure the form isn't actually submitted by returning false */
    return false 
}
</script>

More details about what happens with fetch:

/* this fetches the URL and resolves response */
fetch(`https://www.googleapis.com/customsearch/v1?key=${apikey}&cx=${cx}&q=${input.value}`)

/* this resolves only the responseText */
.then(response => response.text())

/* parse the responseText to Object */
JSON.parse(text)

The Object parsed from JSON looks like this:

Each item in result.items looks like this:

I'll let it up to you figuring out how to add an item to your result DIV. (create an <a> element and set its href attribute to item.link)

Thread Thread
makwelos profile image
Siya

Awesome, thank you so much Jochem.
I got it:)

Thread Thread
cazp83 profile image
César Augusto Zapata

Hi Jochem!

Great tut! I was wondering if It's possible to add the autocomplete feature on this form input

Thread Thread
jochemstoel profile image
Jochem Stoel Author

To use Google Suggest, you need a server. Cross Domain XMLHttpRequests are not allowed. I have actually created an NPM module for Google Suggest that does exactly that. It fetches suggestions from Google Search.
All you need to do is create a server that pipes the response to the client and then use some jQuery auto-complete plugin to render them visually.

Thread Thread
cazp83 profile image
César Augusto Zapata

Hi Jochem, thanks for the advise, very useful. I'll look into this.

Thread Thread
jochemstoel profile image
Jochem Stoel Author

Did you figure it out? Does it need an article?

Thread Thread
cazp83 profile image
César Augusto Zapata

Thanks for the follow up Jochem, however I'm more interested in receiving suggestions based on the sites listed in my search engine rather than the entire web. Recently I found this function (google.search.CustomSearchControl.attachAutoCompletion) so I'm going to try implement it in my project, thanks again.

Collapse
qm3ster profile image
Mihail Malo

Should probably be .catch(console.error.bind(console))

Collapse
jochemstoel profile image
Jochem Stoel Author

That is stupid.

Collapse
qm3ster profile image
Mihail Malo

Suit yourself. What you're doing didn't make it into stable chrome until 2017.

Thread Thread
jochemstoel profile image
Jochem Stoel Author

Where I live it is 2018.

Thread Thread
qm3ster profile image
Thread Thread
jochemstoel profile image
Jochem Stoel Author

Actually since then I thought about this dialogue with you deeply and if that notation was still unsupported so recently then you are absolutely right to point that out.

I was not aware of this, did not consider it either and I am stubborn.

Thread Thread
qm3ster profile image
Mihail Malo

;)

You should see me when I get stubborn.

Collapse
rameshkunwar profile image
rkun

"There are two options for programmatic access. The option Custom Search JSON API is free and has a limit of 10,000 queries per day".

As of 10-12-2019, This Google's site states "Custom Search JSON API provides 100 search queries per day for free. If you need more, you may sign up for billing in the API Console. Additional requests cost $5 per 1000 queries, up to 10k queries per day.

If you need more than 10k queries per day and your Custom Search Engine searches 10 sites or fewer, you may be interested in the Custom Search Site Restricted JSON API, which does not have a daily query limit."

Collapse
imvamckrsna profile image
naga

Can u please guide me a steps how can i implement this in wordpress?

Collapse
jochemstoel profile image
Jochem Stoel Author

No I can not. Implementing it in Wordpress is the same as what you see here only you have to copy paste into your template or a widget. You can do this yourself.

Collapse
imvamckrsna profile image
naga

im confused which template you are referring to i have a JSON api key and the cx id and <async code like this

can u show me the final template that i have to copy paste in searchform.php?

Thread Thread
jochemstoel profile image
Collapse
alphacoffeehacker profile image
alpha-coffeehacker

Hey Jochem,

I'm a total coding nitwit, and I'm a (bit) embarrassed to ask you this, but:
As I'd like to use search more then 10 sites with my GCSE, I want to create a webpage using the API, with search form and the normallooking google-results (as you have made). But really, I can't get it to work.
I'v searched for examples of this, and yours is the most specific I've found so far.
However, I just can't get it to work.
Could you show an example of the code needed to implement this in my googlesites webpage (html, js and css), as you have made for your 'opensource-example'?

Thx a lot!

Collapse
jochemstoel profile image
Jochem Stoel Author

Hey. Follow me here on Dev.to, I will follow you too. Then we can talk through PM and I will gladly help you through the whole process. This keeps this page on point/topic and not side tracked. :)