Building an Intelligent Coffee Order System with Twilio Autopilot

Wouldn't it be great if you could save daily a few, maybe awkward, interactions with other humans and let bots take care of it instead? However, often these bots are not too intelligent when it comes to interacting with you. In this blog post we'll learn how we can build a smarter SMS bot in just a few minutes.

In a previous blog post we talked about how we love to serve coffee at conferences and other events "Twilio-Style" by allowing attendees to order their coffee via SMS. If you haven't read the blog post, make sure to check out Serving Coffee with Twilio Programmable SMS and React. Unfortunately we had a fairly rudimentary way of handling the orders. Until now.

Let's see how we can upgrade this existing project using Twilio Autopilot to be truly intelligent.

If you would prefer to see how this works by watching a video, here is our tutorial on YouTube:

Setup

Before we get started, make sure you have your setup ready. If you've performed the steps in our previous blog post, you are all set. If you haven't, make sure to check out "Serving Coffee with Twilio Programmable SMS and React". Alternatively follow the setup instructions in the README of the Barista Lite GitHub project.

Creating your Personal Coffee Assistant

In order to get started with Twilio Autopilot, we'll have to create a new "Assistant". For this head to the Autopilot section of the Twilio Console and create a new Assistant. You can give the assistant any name. I'll name mine "CoffeeBot" for now.

Once created let's make sure we wire it up to our phone number to start testing the out-of-the-box experience. Click on "Channels" on the left side and choose "Programmable Messaging".

You should see a URL there. Copy it and go to your phone number in the Twilio Console Phone Numbers section and update the "When a message comes in" webhook to the URL you copied and make sure to press Save.

Once saved, take your phone and send any message to your number. You should see as a reply "This is your new Task".

This message comes from the default task that has been created when you created your assistant. But what is a Task?

What are Tasks, Samples, Fields, Models, ...???

When you start working with Autopilot and you haven't worked with any Natural Language Processing before, there might be lots of new terms that might be confusing for you. So let's try to clear these up a bit more.

• Tasks: These are different units of work that you want to perform. Autopilot allows you to perform different "actions" inside a task. Those could be things like "say" to communicate a message to the user, "handoff" to forward the communication to a human, "redirect" to ping a webhook to decide what to do next, "collect" to gather a bunch of data, or many more.
• Samples: In order to let Autopilot understand when to trigger which task we need to provide it with sample statements and map these against the existing tasks. The more samples you have per task, the more intelligently your bot will be able to route your user to the right task.
• Fields: Sometimes your samples are not fully static. For example a sample like "I would like to have one espresso" has some important pieces of info that you want to extract that might vary from user to user. Say the quantity and the type of coffee are actually fields in this sample. We'll later look at how we can work with them and the different available Field Types.
• Models: Every time you modify these parts of your Autopilot assistant, you'll have to rebuild a new model. You can imagine a model as a giant "smart" decision tree that is the outcome of all the info you gave your assistant. The more info you give it, the more complex and smarter your model gets.

Creating your First Task

In order to see your existing tasks and create new ones, click on the "Task Builder" section of your assistant. You'll see one existing task there already called "hello_world". It also has some labels attached to it to signalize that it has been configured as the Fallback, Initiation and OnFailure task. If you want to understand what these all mean or change one of them, click on the "Defaults" tab in the Tasks view.

Instead of creating a task from scratch, let's start by modifying this one. Click on the task and you'll see a code editor pop up that contains the following JSON:

{
  "actions": [
    {
      "say": "This is your new Task"
    }
  ]
}

Inside it you can see it shows the exact text we received earlier. Let's modify it to have a more friendly greeting:

{
  "actions": [
    {
      "say": "Hi there! Thanks for reaching out to your friendly Coffee Bot. Please let me know what you would like to order."
    },
    {
      "listen": true
    }
  ]
}

As you can see we changed the text behind the "say" key but also added a second action called "listen" to tell Autopilot that it should keep the session open. After modifying the JSON, hit save.

Next let's add some samples to trigger this task. Go to the "Natural Language Router" section and click on the expand button next to the text field. This way we can add multiple samples at once by adding them on different lines. Feel free to add whatever sample sentences you want or copy the following:

hey there
hi
what's up
hello

Select the hello_world task next to it and press the "+" button to add them:

After doing all these changes we need to create a new model. Go to the "Build Models" tab and create a new Model Build with a name like "v0.0.1".

Once your model's status changes to "Completed" your bot is ready to be tested.

For this text anything you want to your number and you should see an updated message.

Creating a Dynamic Task

Alright, now that we have a static task let's bring in some more dynamic experiences using Fields. As previously mentioned, Fields allow you to add placeholders inside your samples that will later be automatically extracted by Autopilot so that we can work with them.

Before we can work with the fields though we'll need to create a new task and something that will handle the field values later. For this let's first create a Twilio Function that will log the value of a field and return a say action.

Go to the Functions section of Twilio Runtime and create a new "Blank" function. Give it a name like "Log Field Value" and add a path like /log-value. Change the code of your Function to the following:

exports.handler = function (context, event, callback) {
  console.log(event.Field_Quantity_Value)
  const response = {
    actions: [
      {
        say: "Thank you for your order."
      }
    ]
  }
  callback(null, response);
};

Afterwards go back into your CoffeeBot Assistant and create a new task in Task Builder section. Give it a name like new_order and add the following code:

{
  "actions": [
    {
      "redirect": "https://<your_runtime>.twil.io/log-value"
    }
  ]
}

Make sure to update the URL with the link to your Twilio Function. Afterwards click save to make sure your changes are not lost.

Next we need to add Fields to this task. Click on the "Modify" link next to our task and in the dialog click the "View Fields" link. In there you'll be able to add new Fields. Create a new Field with the name Quantity. Autopilot comes with a variety of predefined data types for Fields. In our case we care about the Quantity number. So go ahead and choose "Twilio.NUMBER" as the type of this field.

Afterwards close the dialog again and switch to Samples as we'll have to add new samples for our task. Expand the input field and place the following values into it:

I would like to have {Quantity} coffee please.
{Quantity} coffee.
Could you make me {Quantity} coffee please?
Can I have {Quantity} coffee?

The {Quantity} tells Autopilot that this is a placeholder for the Field "Quantity". Associate the samples with the new_order task and add them by hitting the "+" button.

After creating the samples, go to the Build Models section again and trigger a new model build. Once that's finished, go back to your Twilio Function to see the logs in the bottom of the Function and take out your phone.

Text it something like "Ten coffee please" and check the logs of your Function. You should see it log "10".

That's because we are logging the parameter Field_Quantity_Value. Autopilot will automatically pass every captured Field as Field_<FIELD_NAME>_Value to the webhook. You can also see that it turned Ten into 10. That's because we told Autopilot that this Field is of type number. It will now handle both numbers as digits or words. Try it by texting "Can I have 15 coffee?"

Adding Custom Field Types

While the built-in Field Types serve a variety of use cases, there will be the situation where you'll want to have your own Field Type. In our CoffeeBot case this would be, for example, the different types of coffee that we serve.

To create a new Custom Field Type go into the Natural Language Router part of your bot and click on the Manage Fields tab. Press the "Create your first Field Type" button and give it a name like coffee_type. Once created click on the name in the list of Field Types and press the Plus button to add new examples. This is where you'll want to add possible values for this type. In our case this would be any valid Coffee type. You can enter one value per line for ease of use.

espresso
americano
latte
mocha
tea
coffee
flat white

Note that the values that you provide here will not be the only valid ones. Autopilot will learn and try to match other words to this field type as well.

After you created them, let's add CoffeeType as a valid Field for our new_order task. Go back into the View Fields part of your new_order Task and add a new Field of name CoffeeType and data type coffee_type.

Now before we build our new model, we need to update our samples to use the new field. Go ahead and delete the old samples for the new_order and instead create new samples using the following value:

I would like to have {Quantity} {CoffeeType} please.
{Quantity} {CoffeeType}.
{CoffeeType}
Could you make me {Quantity} {CoffeeType} please?
Can I have {Quantity} {CoffeeType}?

Once saved, go into Model Builds and build a new Model by incrementing the version in your build name.

Your model will now be able to understand both quantity and different coffee types.

Connecting our Ordering System

To actually be able to create new orders we now need to change the action that we are triggering to our original Twilio Function URL and update our Twilio Function.

Go to your "Barista Create Order" Twilio Function from the previous blog post and update it accordingly:

exports.handler = function (context, event, callback) {
  const ORDER_LIST = 'orders';
  const SERVICE_SID = context.SYNC_SERVICE_SID || 'enter Sync Service Sid';

  const orderType = event.Field_CoffeeType_Value;
  const orderQuantity = event.Field_Quantity_Value || 1;
  const order = `${orderQuantity}x ${orderType}`;

  const orderData = {
    order: order,
    phoneNumber: event.UserIdentifier,
    status: 'open'
  };

  // Create a sync list item for the order
  const twilioClient = context.getTwilioClient();
  twilioClient.sync
    .services(SERVICE_SID)
    .syncLists(ORDER_LIST)
    .syncListItems.create({ data: orderData })
    .then(x => {
      callback(null, {
        actions: [
          {
            say: `Thank you for ordering ${order}`
          }
        ]
      });
    })
    .catch(err => callback(err));
};

This will read the right fields as well as fall back to a quantity of one if no quantity could be detected. Additionally we are using the event.UserIdentifier here to get the phone number since this will be the phone number for SMS and voice bots. Note that this won't work for Alexa or Google Home for example.

Save your Function changes and copy the URL to your Function. Go back into your Autopilot Task and modify the new_order task and update the URL for the redirect to your Function URL.

Make sure you rebuild your model one more time to catch all changes.

If you haven't yet opened your terminal do that now and start your React interface from the previous blog post by running:

npm start

You should see in your browser the following screen:

Let's test if everything is working by texting something like "Can I have two latte please?". Note that we are spelling it lowercase and are altering our sentence from the example sentences.

You should get a confirmation of your order of "2x Latte" and it should appear in the browser UI. Click "Finish Order" or "Cancel Order" and you should receive the respective confirmation via SMS.

What's Next?

This is just the beginning of what you can do with Twilio Autopilot. If you want to learn more, I suggest you check out the Collect action that allows you to do form-filling with ease. You could use it for example to ask the customer for additional information like their name or if they want to have soy milk added to their coffee. Or check out how you can use the Handoff functionality to connect either to a Twilio Flex agent or directly to the phone of your barista for any questions the bot can't answer.

And if you want to import/export your bot to another account for example, you should check out the autopilot-cli that will help you with building your bot.

If you have any questions or if you want to show me what cool thing you built with Autopilot or just in general, feel free to reach out to me:

• dkundel@twilio.com
• @dkundel
• GitHub: dkundel
• dkundel.com