DEV Community

Cover image for Handsfree.js - A web based Face Pointer
Oz Ramos
Oz Ramos

Posted on • Updated on

Handsfree.js - A web based Face Pointer

Liquid error: internal

Handsfree.js is a small wrapper library around web based computer vision libraries that I'm working on for the purpose of helping you interact with web pages handsfree. Currently it only handles head tracking through Jeeliz Weboji, but I also plan to add eye tracking, hand tracking, voice, and full body pose estimation over my next 100 days of code.

I also plan on adding integrations for popular libraries like P5.js, Babylon.js, A-Frame, and others!

But in this post, I'll just focus on how to get a simple Handsfree.js project going which involves three steps:

  • Requiring dependencies
  • Creating a handsfree instance
  • Adding a plugin (a callback that runs on every video inference frame)

At the end, you'll have a red face controlled pointer like in the following:

Liquid error: internal

Adding a handsfree pointer

Because everything is already bundled for you, all you need to do is include the Handsfree.js JavaScript and Stylesheet in your DOM:

<!-- Require dependencies -->
<link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.css">
<script src="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.js"></script>
Enter fullscreen mode Exit fullscreen mode

This adds the Handsfree class to your page, along with basic styles for the pointer. The next step is to create an instance of Handsfree:

const config = {}
const handsfree = new Handsfree(config)
Enter fullscreen mode Exit fullscreen mode

You'll need one instance for each camera that you plan to use (if you have multiple cameras), but each instance can track multiple users. See the documentation on the config object for the defaults and other settings you can use.

And that's it! To start and stop tracking, use handsfree.start() and handsfree.stop().

Adding functionality, and using the pointer values

On each frame, your handsfree instance will have several properties that contain useful information:

// The x coordinate of the pointer on the screen
handsfee.head.pointer.x

// The y coordinate of the pointer on the screen (from 0 at the top)
handsfee.head.pointer.y

// The pointer element
handsfree.head.pointer.$el

// The pointer state ("", "mouseDown", "mouseDrag", "mouseUp")
handsfree.head.pointer.state

// The head position [x, y, scale]
handsfree.head.translation

// The head rotation [pitch, yaw, roll]
handsfree.head.rotation

// Head Morphs (how raised the eyebrows are, how smiley your smiling to each side, etc)
// @see https://github.com/handsfreejs/handsfree/wiki/Head
handsfree.head.morphs
Enter fullscreen mode Exit fullscreen mode

The Handsfree class has a global loop for all instances which you can hook into with the use method:

// Create a simple plugin that displays pointer values on every frame
Handsfree.use('consoleLogger', (instance) => {
  console.log(instance.head.morphs)
})

// Same plugin, but with destructuring
Handsfree.use('consoleLogger', ({head}) => {
  console.log(head.morphs)
})
Enter fullscreen mode Exit fullscreen mode

These are called plugins, where "consoleLogger" is the name of the plugin and instance is the handsfree instance running the plugin (handsfree = new Handsfree()). Adding multiple plugins with the same name overwrites the previous plugin, and to disable a plugin you can call handsfree.stop().

Handsfree.js ships with a few plugins - "head.click" and "head.vertScroll" - which add clicking functionality (with a smile gesture) and scrolling as in this tweet:

Liquid error: internal

A complete example

<!DOCTYPE html>
<head>
  <!-- Require dependencies -->
  <link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.css">
  <script src="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.js"></script>

  <style>
    button {font-size: 24px; padding: 20px}
  </style>
</head>
<body>
  <!-- Let's always ask the user to start -->
  <button onclick="handsfree.start()">Start Webcam</button>
  <button onclick="handsfree.stop()">Stop Webcam</button>

  <script>
    // Create a new instance. Use one instance for each camera
    const handsfree = new Handsfree({})

    // Create a simple plugin that displays pointer values on every frame
    Handsfree.use('consoleLogger', ({head}) => {
      console.log(head.rotation)
    })
  </script>
</body>
Enter fullscreen mode Exit fullscreen mode

Run the above code on Glitch.

Next Steps

I hope this gave you a good introduction into Handsfree.js. In this post I covered how to setup a basic pointer, and in future posts we'll be going over:

  • How to use head morphs (eyebrows, smiles, kissy faces, and more)
  • Get the users head pose (yaw, pitch, roll)
  • Multiplayer support
  • Complex plugins
  • Integrations with popular frameworks
  • and more

Follow me on Twitter at @HeyOzRamos if you'd like to follow along on my 100DaysofCode. Thanks for reading!
Oz

Updates

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.