How Visual Similarity and Custom Metadata can Enhance Your Search

Data can be thought of simply as a thing we want to remember for future use. Metadata helps describe the data we’re trying to remember. So when we are putting it in context of visual similarity, we are adding metadata to describe inputs (data) that may not be visually distinguishable. Here’s what that means for searching images!

When we say metadata at Clarifai, this describes some attribute relating to a particular image that may or may not be visibly tangible. Think about it as where we could store all of the information we want the image to hold onto that is valuable to our project or business. Here are a few examples:

• We could have users searching through a catalog and we want to record which product they viewed. We would be able to reference metadata of the product’s ID without having to make a callback to our database.
• We could add metadata to our inputs so that when a user comes along and wants to find an item nearby, we filter based on a zip code or region.

We can craft Custom Metadata to suit any need.

Let’s go through an example in-depth. We’re going to look at the case of a shoe store and how we can search using an image and our metadata. We will be able to find items that are visually similar to what we want and also filter items based upon them being on sale. If you want to see all of the code already written up, you can check out this GitHub repo and its README.

Requirements

If you haven’t already, make sure to sign up for a free account on Clarifai and create an application. We will also need to be sure to have NodeJS installed.

We have a rather short list of data for our little footwear shop. It is a CSV with several columns for each of the data points we want to represent: Product ID, Type, Color, Brand, Price, On Sale, In Stock, Image Source. Let’s use the prebuilt data here for our example and save it to our project folder as shoe-data.csv.

Convert CSV and upload to Clarifai

The team over at Adaltas decided to share their code to help people parse data from spreadsheets in Node. It allows for flexibility of our data, so give them some kudos. We will install it with npm install --save csv-parse in Terminal. Before this data is useable we need to convert each row over to a JSON Object. Open up a new file upload.js and we will write all of our actions here:

/* upload.js */
const parse = require('csv-parse')
const fs = require('fs')

// ~Hidden magic we will come back to~

fs.readFile(__dirname+'/shoe-data.csv', 'utf8', (err, data) => {
  if(err) { return console.log(err) }
  parse(data, { columns: true }, (err, output) => { 
    if(err) { return err; }
    shoeData = output.map((shoe) => { return convertData(shoe) })
    uploadInputs(shoeData);
  });
})

We still have the hidden magic to fill in that explains our convertData() and uploadInputs()! We have to start with adding in the code that will let us use Clarifai. To install the Clarifai JavaScript client, go to Terminal and write npm install clarifai --save. Include the client with const Clarifai = require('clarifai') right below the other modules. Be sure to place our API Key for the application:

/* upload.js */
const parse = require('csv-parse')
const fs = require('fs')
const clarifai = require('clarifai')

const app = new Clarifai.App({ apiKey: 'YOUR_API_KEY' })

// ...

Now we want to be sure to write out the convertData() and uploadInputs() functions. convertData() will take our results from reading the CSV and then convert the data into inputs with metadata. uploadInputs() then takes that data and sends it to Clarifai for it to store.

/* upload.js */
const parse = require('csv-parse')
const fs = require('fs')
const Clarifai = require('clarifai')

const app = new Clarifai.App({ apiKey: 'YOUR_API_KEY' })

const uploadInputs = (inputs) => {
    app.inputs.create(inputs).then(
      // Success
      (response) => { console.log('Successful Upload!') },
      // Error
      (error) => { console.error(error) }
    )
}

const convertData = (data) => {
  return {
    url: data.imageSrc,
    metadata: data
  }
}
// ...

With us running node upload.js we have everything handled for taking a CSV and uploading it onto Clarifai.

Searching on Clarifai

Let’s write another small script that will perform our search which we will cleverly name search.js. We will use this image and then apply metadata for items on sale as TRUE. Our image would filter what item a user would want to have. This is how we incorporate visual similarity into our search. However, there is no visual cue in the image alone to find out which items are on sale. That’s a great reason why we need to apply the metadata.

/* search.js */
const Clarifai = require('clarifai')

let app = new Clarifai.App({ apiKey: 'YOUR_API_KEY' })

// Searching by visual similarity
app.inputs.search([
    {
      input: {
        url: 'https://farm4.staticflickr.com/3370/3344620504_b547190891_o_d.jpg'
      }
    },
    {
      input: {
        metadata: {
          sale: 'TRUE'
        }
      }
    }])
.then((response) => {
  response.hits.map(
    (hit) => {
      console.log(`Price: $${hit.input.data.metadata.price}USD; URL: ${hit.input.data.image.url}`)
  })
})

We can run it with node search.js in Terminal. We get a list to all of our prices and the URLs associated with the items. An important note is that our metadata field is sensitive to how the data is searched. If we have capitals in any of our keys or values, we need to be sure that they match exactly when we go to search. Otherwise, they will consider the object you put in and the object you wanted to find as different things.

Conclusion

We’ve just played around with one way to use metadata. Remember, we added a bunch of different labels onto our inputs. Some metadata may be more valuable than others but it is entirely freeform. If you are curious about some more search features such as by geo location, public concepts or anything else, read more about it all in our guide. Let us know if you need any help or how you make use of custom metadata at hackers@clarifai.com!

The post How Visual Similarity and Custom Metadata can Enhance Your Search appeared first on Clarifai Blog | Artificial Intelligence in Action.