In a constantly changing world where powerful AI-based applications are day by day more accurate and efficient than any technology, it’s hard to keep track of such a big transformation.
As developers, we need to learn how to leverage the technology in such a way to solve problems from the real world and even if we are not implementing some of the world’s most cutting edge technologies we can integrate them into our daily workflows.

My eagerness to try out the latest technology and the love for coffee made me search for a solution of a problem I had, how do I check if I have milk in the fridge? (yes, I make my own cappuccino).
Of course, many exciting systems came into my mind, like a bunch of pressure sensors placed into the fridge or a cam so I can read real-time what items do I own. But this comes with the obvious risk of not implementing it since it introduces the hardware part so the complexity grows.

So back to my need I started to implement an app for the Google Assistant, an app to help me keep track of the items I have into the fridge.

But what are actions?

Actions on Google lets you extend the functionality of the Google Assistant with Actions. Actions let users get things done through a conversational interface that can range from a quick command to turn on some lights or a longer conversation, such as playing a trivia game.
There are many conversation design principles but the main idea is to create such a natural conversation with the Assistant in a way that the user forgets that he is not speaking with a human.

Goal

My goal is to have a fast and easy way to check if I have milk. But why not extend this to any item that can be placed into the fridge and have sort of a “voice list” to quickly access it through voice using Google Assistant.

I started writing sample dialogs.
Sample dialogs are the key to creating great Actions on Google; they’ll give you a quick, low-fidelity sense of the “sound-and-feel” of the interaction you’re designing.
They convey the flow that the user will actually experience, in my case a way to check if there is any milk into the fridge.

During the conversation sketch, I was thinking of a possibility for the users to add any item (milk, eggs, etc) into the fridge, remove one or check what the fridge contains using only speech.

Going deeper into any possible conversation that I can have with the application, I ended in some flows that are hard to follow on a device without any screen capability (like the google home mini).

Available devices:

For example, if I’m going to check all the items I have into the fridge and let’s say that I stored like, 20 or 30 items, to keep track of them by voice is hard. I need to adapt my spoken conversation into a multimodal conversation and check if there is a display available and build a list with all the items.
In this way, I scale my application across multiple devices (spoken alone or spoken with display).
So for devices with an available display, I built a list:

Development part

Building your own natural language processing service can be challenging, so using Dialogflow is the best choice.
First I created a developer project for Actions on Google Console.
Second I created a Dialogflow agent where I started to define all my Actions (add, remove or check fridge items). To create the conversation, here is the place to define intents, to create conversation's grammar or the things users can actually say to the Action.

Intents I've created:

The backend part I wrote it in Kotlin. I use Firebase Firestore to save all users items and query for them when “check food” intent is matched.
I deployed the application into the Google Cloud Platform. This way I do not worry about managing the infrastructure and how to scale it.

Final Thoughts

No matter what programming language you know, no matter what day by day projects you are working on, all the time look forward to solving a need. The technological range is quite big as well as the efficiency and reliability of the technology vary but this should not keep you from doing it. Maybe you will not revolutionize the future but definitely you are having fun trying it.

You can check the app while talking with your Google Assistant, <Ok, Google, Ask Smarty Fridge to check my food.>

Let’s shape the future together.
Thank you for reading this. If you have any questions, comments or funny jokes comment below.