Oz Ramos

Posted on Jan 2, 2021

Introducing Handsfree.js - Integrate hand, face, and pose gestures to your frontend 🖐👀🖐

#javascript #webdev #handsfree #machinelearning

Hello and thank you for checking out this introductory post, I'm so excited to share Handsfree.js with you! Handsfree.js is a client side library that helps you add hand, face, and pose estimation to your front end projects in a snap ✨👌

Since this is an introductory post, I'll start by sharing some of the things I've made with it so that you can get an idea of what's possible. Once I've hopefully hyped you up a bit I'll then show you how to get started!

Handsfree.js can help you do quite a bit, and I'm using it to completely and totally handsfree-ify the web and therefore entire world around us. You can see how I plan to do that in my Master Plan or you can see me actually doing it on Twitter @Midiblocks.

OK! Let me show you what you can do with Handsfree.js ✨

Examples

Use it to trigger events

Just yesterday I released the Pincher Plugin which emits 24+ pinching events with 3 states - start, held, released - for pinching with your index, middle, ring, and pinky fingers. It's modelled after the Mouse Events and you can listen to them similarly with document.addEventListener(): https://handsfree.js.org/ref/plugin/pinchers.html

Use it to scroll pages handsfree

Here's a Browser Extension I'm working on that helps you scroll websites handsfree. It uses the MediaPipe Hands model to track your hands. This GIF was actually really easy to make with the built in pinchScroll plugin, which enables this customizable functionality in a single line of code: https://handsfree.js.org/ref/plugin/pinchScroll.html

Use it to create new kinds of Assistive Technologies

This is one of my favorites, and it uses the "Face Pointer" plugin to allow you to move a pointer with your face, scroll pages, and click on things. It's powered by the Jeeliz Weboji model and a few face plugins.

Use it to control desktop games

Here's me playing "Into the Breach" on my desktop with Face Pointers and Hand Pointers. These are super easy to make too, all I did was use the Face and Hand Pointer plugins and then stream them over to my desktop with Robot.js to trigger native mouse events:

Use it to make your own games

But why just play games when you can make them too!? Here are a few games I've made, which I plan on grouping together into a "Mario Party" like game where you roll dice to move on a board and then play these minigames with your friends at the end of each round.

Here is "DuckFace Hunt", "Flappy Pose", and "Handsfree Jenga":

Use it to control robots and drones

Of course, you're not limited to controlling things on the web or even desktop. With WebSockets you control anything connected to your computer, like this Universal Robot which I tried to puppeteer with my own head:

Use it for art, music, and other experiences

There's so much more that you can do! Here are some other experiments like my upcoming "Diffusionist" app designed to help you make trippy art to the beat of music (check out my brand new Instagram for the audio version). I'm also making a WebXR DevTools Chrome Extension so that you can work on WebXR apps handsfree without XR equipment:

Getting Started

Great! So now that I've shown you a little of what you can do, let me show you how. Don't worry if this is overwhelming at first, it's more of an overview. I'll have lots of shorter and more focused tutorials coming soon 🙏

If you clone my repo (and please give it a star 🤗) you can find a boilerplate in /boilerplate/cdn.html. I'll have many more soon 😊

Initializing and starting Handsfree

The easiest way to get started is with a CDN. If you'd like, you can create an HTML file and copy/paste this in without the need for a server:

<head>
  <!-- Import helper classes and styles -->
  <link
    rel="stylesheet"
    href="https://unpkg.com/handsfree@8.1.1/build/lib/assets/handsfree.css" />
</head>
<body>
  <!-- Import Handsfree.js in body (as it adds body classes) -->
  <script src="https://unpkg.com/handsfree@8.1.1/build/lib/handsfree.js"></script>

  <script>
    // Use the hand with defaults (and show the webcam with wireframes)
    handsfree = new Handsfree({
      showDebug: true,
      hands: true
    })

    // Start webcam and tracking (personally, I always like to ask first)
    handsfree.start()
  </script>
</body>

You can also import with NPM. By default this will still load the models from a CDN as they are quite large (some are over 10Mb), but I have instructions for ejecting the models into your assets folder here: https://handsfree.js.org/#hosting-the-models-yourself

npm i handsfree

handsfree = new Handsfree({
  showDebug: true,

  // Use the hand model with custom config
  hands: {
    // Always make sure to enable them
    enabled: true,

    // Let's track up to 4 hands. It's best to be kind and ask permission first tho!
    maxNumHands: 4,
  }
})

// Start webcam and tracking (personally, I always like to ask first)
handsfree.start()

For the full list of config options you can pass into Handsfree, see: https://handsfree.js.org/ref/prop/config.html#the-full-list

Working with the data

Of course that'll just show the wireframes over your hands, but it won't actually do anything yet. There are two main ways to work with Handsfree.js, and my preferred way is by creating plugins using handsfree.use(newPluginName, callback). I call them plugins because they "plug into" the main webcam loop that's started when you run handsfree.start().

Plugins run their callback on every webcam frame and receive all the data from all the running computer vision models. Here's a very simple plugin that simply console logs data. I'll call it "logger":

// Let's use our hands again
handsfree = new Handsfree({showDebug: true, hands: true})
handsfree.start()

// Let's create a plugin called "logger" to console.log the data
handsfree.use('logger', (data) => {
  // I like to always bail if there's no data,
  // which might happen if you swap out hands for the face later on
  if (!data.hands) return

  // Log the data  
  console.log(data.hands)

  // Do something if we are pinching with left [0] pinky [3]
  if (data.hands.pinchState[0][3] === 'held') {
    console.log('pinching with left pinky')
  }
})

Once you create a plugin, it becomes available at handsfree.plugin.pluginName and comes with a few methods and properties. Most importantly, they get a .disable() and .enable() method:

handsfree.plugin.logger.enable()
handsfree.plugin.logger.disable()

// This is what the callback gets mapped to,
// and is what gets called on every frame that this plugin is enabled
handsfree.plugin.logger.onFrame

If you need more advanced functionality then you can pass an object with specific hooks that'll run during various phases of the plugin. For example:

handsfree.use('advancedLogger', {
  // True by default
  enabled: true,

  // A list of strings for tagging this plugin.
  // Later you can bulk disable/enable these with: handsfree.enablePlugins(['tag1', 'tag2'])
  tags: [],

  // This special property can be adjusted later (or even before!) in various ways
  config: {},

  // Called immediately after the plugin is added, even if disabled
  // The `this` context is the plugin itself: handsfree.plugin.advancedLogger
  // If you need to create DOM elements or other setup, this is the method to do it in
  onUse () {},

  // Called when you .enable() this plugin
  onEnabled () {},
  // Called when you .disable() this plugin
  onEnabled () {}
})

Using data without plugins

Sometimes you may only want to track just one frame, or use an image, canvas, or video element instead of a webcam, or you might be in a part of your app where you don't easily have access to your handsfree (like in node module). In these cases, you can just listen to events on the document.

Because these are events, the data you want is always in ev.detail.data

// This will get called on every frame
document.addEventListener('handsfree-data', ev => console.log(ev.detail.data))

// Listen to when the thumb and index (0) are pinched on any hand
document.addEventListener('handsfree-finger-pinched-0')

// Listen to when the right (1) thumb and pinky (3) are pinched
document.addEventListener('handsfree-finger-pinched-1-3')

Also, know that you can always access the data directly on your handsfree instance:

console.log(handsfree.data.hands)

Updating models and plugins

The real magic of Handsfree.js is in its ability to instantly swap out models and plugins. This is useful if different routes in your app have different handsfree user experiences. This is where the very powerful handsfree.update(config) comes into play. I use this everywhere on Handsfree.js.org to allow you to try out different demos without restarting the webcam.

handsfree.use takes in the same Config Object as when you instantiate Handsfree, but it does a few extra things:

It stacks changes, so if you only pass in handsfree.update({facemesh: true}) while you have hands turned on then you'll end up with both
It automatically handles loading in any required models and dependencies
Gives you the ability to also configure plugins, or turn them off completely

Here's an example

// Start with hands
const handsfree = new Handsfree({hands: true})
handsfree.start()

// Add facemesh
handsfree.update({facemesh: true})

// Replace both with pose
handsfree.update({
  hands: false,
  facemesh: false,
  pose: true
})

// Use Weboji and enable the Face Pointer plugins
handsfree.update({
  hands: false, facemesh: false, pose: false,
  weboji: true,

  plugin: {
    // Enable some plugins
    faceClick: true
    faceScroll: true,
    // Update the special .config properties of the plugins (this is so magical!)
    facePointer: {
      speed: {
        x: 2,
        y: 2
      }
    },
  }
})

You can also bulk disable and enable plugins, even by tag:

// Disable and enable them all
handsfree.disablePlugins()
handsfree.enablePlugins()

// Disable all the "core" tagged plugins
handsfree.disablePlugins('core')

// Enable handsfree "browser" tagged plugins
handsfree.enablePlugins('browser')

Each of the current plugins has a set of configs that you can update in real time. Just access their .config by name:

// Change the Face Pointer speed
handsfree.plugin.facePointer.config.speed.x = 2
handsfree.plugin.facePointer.config.speed.y = 2

// Set the threshold for how much you have to smile to click (0 - 1)
handsfree.plugin.faceClick.config.morphs[0] = .25

Classes

Anytime something happens within Handsfree.js, a class is added to the document.body. This makes styling your app or showing and hiding elements based on the state super easy. Here are where you can find them:

Generic classes: https://handsfree.js.org/ref/util/classes.html
Pinching states: https://handsfree.js.org/ref/plugin/pinchers.html#classes

Thanks for reading 🙏

So that about covers the basics! I know that might have been overwhelming, but one of my New Year Resolutions is to write more and so I'll be releasing smaller, more focused tutorials about once a week covering one specific topic at a time.

I start my 2nd residency at the STUDIO for Creative Inquiry this month where I'll be exploring new ways to apply Handsfree.js. It's my full time thing so please leave me a comment below or stop by my Discord so that I can help you integrate Handsfree.js in new and creative ways.

If you'd like to learn a little bit about my story and what inspired me on this path 3 years ago, do check out my GitHub Sponsors. Thank you so much for reading this introduction, I can't wait to see what you do with Handsfree.js 🖐👀🖐

Top comments (38)

Shailesh Vasandani • Jan 3 '21

Amazing project! Super excited to see how this can be used for accessibility — I see it opening up entirely new avenues of interacting with the web, especially if paired with something like VR. The possibilities are endless!

Thanks so much for sharing!

Oz Ramos • Jan 3 '21

Thank you! Accessibility is actually what initially inspired this project. There are many nights when I have a hard time falling asleep because of all the possibilities lol and by releasing this as a library and browser extension I'm hoping to provide the tools to help others explore them!

Brayden W ⚡️ • Jan 3 '21 • Edited

Looks awesome! and a bit familiar... 😉

🤯

Oz Ramos • Jan 3 '21

100% what I think about every time I sit down to code haha

Brayden W ⚡️ • Jan 3 '21

😂😂

Aiden Bai • Jan 2 '21

I tried out handsfree around dec of 2019. Easily one of the coolest js libraries out there!

Oz Ramos • Jan 2 '21

Oh thank you so much! I ended up taking a break for almost a year 😅 but the break has allowed me to recharge and take this in a new and exciting direction

Now I'm solely focused on the journey and not the destination, and I've been having sooo much fun making it! Thanks for the comment :)

Tyler Carroll • Jan 20 '21

I’m a noob who will be building a portfolio in the next few months and I would LOVE to have a project in there made with handsfree.js! However, idk what would be plausible for someone with limited experience to make. Any ideas? Amazing lib btw!!!

Oz Ramos • Jan 21 '21

Hi that would be amazing and also great question! The thing is there are so many applications so it's hard to suggest something 😅 how I go about it is to just making something as normal (where you can use with mouse and keyboard) and then make it handsfree...my next batch of plugins will emit mouse and keyboard events, so you wouldn't even really need to do much with Handsfree.js other than to listen to the events in your app as you would any other mouse or keyboard event

I know how vague that response was 😅 I'll end my next series of posts with an "Ideas" section to help you pick a direction to try and I'll include more boilerplate code and samples.

Thanks for the question and have fun with your portfolio!

Thomas Bnt ☕ • Jan 2 '21

Badass 🤩🤩🤩

Oz Ramos • Jan 2 '21

Thanks! I have lists upon lists for follow up tutorials, can't wait!

Thomas Bnt ☕ • Jan 2 '21

Greaat ! You can notify me if you want when you posted a new tutorial 😃

Oz Ramos • Jan 2 '21

That's super encouraging, thank you! I'll make a few first and then ping ya 🙏

akbartus • Feb 9 '21

Oz thank you for great solution for tracking. I wonder is it also suitable for mobile based experiences. I already tried MediaPipe hands before and it was not mobile friendly meaning it was supporting well only latest models of phones with bigger Ram. On older devices there was lag. Is handsfree good also for mobile web (Chrome, Mozilla)?

tq-bit • Jan 3 '21

That's easily one of the coolest project I've ever seen.

Jen Looper • Jan 20 '21

wow, this is incredible. Looking to do some hand shadows with this, great that you can have two hands showing!

Sean Allin Newell • Jan 3 '21

WOWOW!! Big major stonks coming your way.

Alex Chiu • Jan 3 '21

This is amazing!!!

Naowal Siripatana • Jan 3 '21

Awesome, It can.expand to.many thing.

nikhilmwarrier • Jan 3 '21 • Edited

Badass! Really cool! I am pretty darn sure it will be the jQuery of the AR/VR web...

Jake Tracey • Jan 3 '21

This is incredible! Going to do some proof of concepts for users with limited mobility using Face pointer!

Oz Ramos • Jan 3 '21

Thank you and please do! Here is the documentation for my current implementation of the Face Pointer along with the different configs. At the bottom is the current code for it if you'd like to see how it's done: handsfree.js.org/ref/plugin/facePo...

I'll be re-implementing the automatic face pointer calibrator from the last version soon. By default it works best when the webcam is directly in front of you but the calibrator allows it to be used from different positions (ex: when the webcam is attached to the arm of a wheelchair).