DEV Community

Cover image for Tutorial: Hacking a robot while respecting fundamental laws of robotics
Nicolas Greniรฉ for Typeform

Posted on

Tutorial: Hacking a robot while respecting fundamental laws of robotics

This past week I got invited to attend Twilio Signal conference in San Francisco as part of their Champions program.

I had the chance to meet other awesome Champions at our Summit. You should all follow them, they are all doing amazing things in their communities all around the world. Find more about the Twilio Champions program over here.

The conference itself was full of great announcements around the Twilio eco-system. I am mostly excited by Conversations, as a way to connect many people uing different systems (SMS, WhatsApp, chat...) into the same conversation. The new Twilio CLI to do everything around Twilio directly in the terminal looks pretty awesome too.

Another big announcement at Signal was version 3 of TwilioQuest available on Desktop. If you are not familiar with TwilioQuest, imagine a video game where you gain XP points by solving coding challenges.
It's a fun way to learn about the Twilio world and programming in general.

screenshot of TwilioQuest

Screenshot of TwilioQuest

The main character you interact within TwilioQuest, is named Cedric and is a friendly robot that is guiding you through your quest against the Legacy Systems.

The thing I did not realize: Cedric is real! He and his friends attended Signal too. They were so many roaming the expo floor and booths.

There even was a special Signal mission on TwilioQuest to earn XPs if you were spending some time with the team of Misty Robotics (parent company of the Misty robot, Cedric's robot family).

And this exactly what I did! I went over to meet the Misty Robotics team to learn more about those friendly robots๐Ÿค–

The team was organizing Developers testing sessions, and inviting people to try their Hello World tutorial to "hack" the robot.

Misty is an incredible robot, among many things she has multiple sensors to move around your house without bumping into anything, a speaker to express herself, a screen to show some emotions, and a camera with face recognition capability. And all that is hackable through an SDK! ๐ŸŽ‰

Misty robot look

Isn't she lovely?

I had a lot of fun following the tutorial. I made Misty move, make her blink her light and even taught her to recognize me. At the end of the session, the team was handing out surveys to collect feedback. And as you may have already guessed it was a Typeform! ๐Ÿคฉ

Then I had no option... I had to connect Misty to Typeform.

I wanted to make Misty react every time someone was filling up the survey.

Here is how I managed to connect the two.

Overall principle ๐Ÿ“–

We rely on Typeform webhooks functionality to react immediately after someone answered.

The webhook is received by an app that can call the Misty API and make the robot react.

But to be able to communicate with the Misty robot, your app needs to be on the same WIFI network, so it's most likely that you will run it locally on your laptop.

We will use ngrok to expose localhost to the world ๐ŸŒ.

Make the Misty move her arm

We create a small express app with only one endpoint in a file named app.js. This endpoint will receive webhook notification and then call Misty API.

This is how it looks:

const express = require('express')
const app = express()
const port = 3000

const axios = require('axios')
const bodyParser = require('body-parser')

// parse application/x-www-form-urlencoded
app.use(bodyParser.urlencoded({ extended: false }))

// parse application/json

const ROBOT_IP = '' //change to your own'/', (req, res) => {

    method: 'POST',
    url: `http://${ROBOT_IP}/api/arms`,
        Arm: "left",
        Position: -90,
        Velocity: 100,
        Units: "degrees"

app.listen(port, () => console.log(`Example app listening on port ${port}!`))
Enter fullscreen mode Exit fullscreen mode

You can run the app by using the command node app.js

And to make her arm go up, run the following command curl -X POST https://localhost:300 in your terminal.

And if everything works well you should see Misty's left arm going up ๐ŸŽ‰

Connect it to Typeform ๐Ÿ”—

All this is working well locally, but wouldn't it be cool to connect it to other services?

To make it happen we need the help of an awesome tool called ngrok. ngrok will expose your localhost and make it available to rest of the internet by giving it a URL. This URL is unique to you and every time you launch ngrok. Make sure to read its getting started guide.

Once you have ngrok installed you can run the command ngrok http 300.
This should give you a URL back. We are going to use it to call our app from the outside.

Screenshot of ngrok

Screenshot of ngrok

On your Typeform dashboard, we can now select the form we want to connect to Misty, and under Connect > Webhooks add a new webhook with this URL.

And voilร  ๐ŸŽ‰
You now have a robot that raises her arm every time someone fills up your typeform.

This will work until you kill ngrok or the node app. Ngrok will give you a new URL every time you launch it, so just keep it open while you are developing.

Go beyond ๐Ÿš€

After seeing Misty raising her arm the first time, I could not restrain myself, and I shouted a loud "Woo-hoo" on the conference floor ๐Ÿ˜Š.

But I could not stop jus now when I was so close to building something even cooler.

My ultimate goal was to build an app that will make Misty speak and react to the things that were posted on the typeform.

Make her arm move dynamically ๐Ÿ‘‹

So I built a simple typeform, where people could leave their name and decide which arm Misty should raise.

To make it easier to extract data from typeform webhook payload, I had to update the ref of my questions. On your typeform select the question and look at the bottom of the sidebar and click Edit. There you can change it to whatever you please. I called mine username and arm_choice.

screenshot typeform edit ref

Edit Ref on typeform question

Now let's see how to extract those values from the webhook payload ๐Ÿ‘‡

  const { form_response } = req.body
  const {answers} = form_response

  let arm_field = answers.find((a) => a.field.ref === 'arm_choice')
  let selected_arm = arm_field.choice.label

  let username_field = answers.find((a) => a.field.ref === 'username')
  let username = username_field.text
Enter fullscreen mode Exit fullscreen mode

Then you can pass select_arm value to Misty API, and raise the corresponding arm.

    method: 'POST',
    url: `http://${ROBOT_IP}/api/arms`,
        Arm: selected_arm,
        Position: -90,
        Velocity: 100,
        Units: "degrees"
Enter fullscreen mode Exit fullscreen mode

Maker her speak ๐Ÿค–๐Ÿ—ฃ๏ธ

Misty REST API lets you send WAV audio files using the SaveAudio endpoint (doc) but... they have to be encoded in base64 ๐Ÿ˜ค

So I had to find a hack to generate a .wav file and convert it to base64.

It took me a couple of trials to find the perfect library to do it, but say.js saved my day!

const say = require('say')
say.export(`Hello ${username}`, 'Alex', 0.75, `hello_${username}.wav`, (err) => {
  if (err) {
    return console.error(err)
Enter fullscreen mode Exit fullscreen mode

Using those few lines, it creates a .wav file named hello_${username}.wave with a voice that says Hello nicolas for example.
Unfortunately, I could not find the name for the female voices on my mac, so we are stuck with Alex for now ๐Ÿคทโ€โ™‚๏ธ

cont fs = require('fs');
let wav = new WaveFile(fs.readFileSync(`hello_${username}.wav`));
let wav64 = wav.toBase64()

    method: 'POST',
    url: `http://${ROBOT_IP}/api/audio`,
      FileName: `hello_${username}.wav`,
      Data: wav64,
      ImmediatelyApply: true, //make Misty play it right away
      OverwriteExisting: true
Enter fullscreen mode Exit fullscreen mode

And this ๐Ÿ” is how I read the file, convert it to base64 and call Misty API to make it play on the robot.

And this how it finally looks ๐Ÿคฉ
Liquid error: internal

The code for the entire app is available here๐Ÿ‘‡

Hope that got you excited to hack things around and play with Misty ๐Ÿค–.
Their crowdfunding campaign is over, but you can already pre-order it.

๐Ÿ™ Special thanks to Twilio Champions team for inviting me to be part of this great adventure ๐Ÿ˜ƒ

Discussion (0)