DEV Community

loading...
Cover image for Handsfree.js - A web based Face Pointer

Handsfree.js - A web based Face Pointer

checkboxoz profile image Oz Ramos ・Updated on ・3 min read

Handsfree.js is a small wrapper library around web based computer vision libraries that I'm working on for the purpose of helping you interact with web pages handsfree. Currently it only handles head tracking through Jeeliz Weboji, but I also plan to add eye tracking, hand tracking, voice, and full body pose estimation over my next 100 days of code.

I also plan on adding integrations for popular libraries like P5.js, Babylon.js, A-Frame, and others!

But in this post, I'll just focus on how to get a simple Handsfree.js project going which involves three steps:

  • Requiring dependencies
  • Creating a handsfree instance
  • Adding a plugin (a callback that runs on every video inference frame)

At the end, you'll have a red face controlled pointer like in the following:

Adding a handsfree pointer

Because everything is already bundled for you, all you need to do is include the Handsfree.js JavaScript and Stylesheet in your DOM:

<!-- Require dependencies -->
<link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.css">
<script src="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.js"></script>

This adds the Handsfree class to your page, along with basic styles for the pointer. The next step is to create an instance of Handsfree:

const config = {}
const handsfree = new Handsfree(config)

You'll need one instance for each camera that you plan to use (if you have multiple cameras), but each instance can track multiple users. See the documentation on the config object for the defaults and other settings you can use.

And that's it! To start and stop tracking, use handsfree.start() and handsfree.stop().

Adding functionality, and using the pointer values

On each frame, your handsfree instance will have several properties that contain useful information:

// The x coordinate of the pointer on the screen
handsfee.head.pointer.x

// The y coordinate of the pointer on the screen (from 0 at the top)
handsfee.head.pointer.y

// The pointer element
handsfree.head.pointer.$el

// The pointer state ("", "mouseDown", "mouseDrag", "mouseUp")
handsfree.head.pointer.state

// The head position [x, y, scale]
handsfree.head.translation

// The head rotation [pitch, yaw, roll]
handsfree.head.rotation

// Head Morphs (how raised the eyebrows are, how smiley your smiling to each side, etc)
// @see https://github.com/handsfreejs/handsfree/wiki/Head
handsfree.head.morphs

The Handsfree class has a global loop for all instances which you can hook into with the use method:

// Create a simple plugin that displays pointer values on every frame
Handsfree.use('consoleLogger', (instance) => {
  console.log(instance.head.morphs)
})

// Same plugin, but with destructuring
Handsfree.use('consoleLogger', ({head}) => {
  console.log(head.morphs)
})

These are called plugins, where "consoleLogger" is the name of the plugin and instance is the handsfree instance running the plugin (handsfree = new Handsfree()). Adding multiple plugins with the same name overwrites the previous plugin, and to disable a plugin you can call handsfree.stop().

Handsfree.js ships with a few plugins - "head.click" and "head.vertScroll" - which add clicking functionality (with a smile gesture) and scrolling as in this tweet:

A complete example

<!DOCTYPE html>
<head>
  <!-- Require dependencies -->
  <link rel="stylesheet" href="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.css">
  <script src="https://unpkg.com/handsfree@6.0.1/dist/handsfreejs/handsfree.js"></script>

  <style>
    button {font-size: 24px; padding: 20px}
  </style>
</head>
<body>
  <!-- Let's always ask the user to start -->
  <button onclick="handsfree.start()">Start Webcam</button>
  <button onclick="handsfree.stop()">Stop Webcam</button>

  <script>
    // Create a new instance. Use one instance for each camera
    const handsfree = new Handsfree({})

    // Create a simple plugin that displays pointer values on every frame
    Handsfree.use('consoleLogger', ({head}) => {
      console.log(head.rotation)
    })
  </script>
</body>

Run the above code on Glitch.

Next Steps

I hope this gave you a good introduction into Handsfree.js. In this post I covered how to setup a basic pointer, and in future posts we'll be going over:

  • How to use head morphs (eyebrows, smiles, kissy faces, and more)
  • Get the users head pose (yaw, pitch, roll)
  • Multiplayer support
  • Complex plugins
  • Integrations with popular frameworks
  • and more

Follow me on Twitter at @HeyOzRamos if you'd like to follow along on my 100DaysofCode. Thanks for reading!
Oz

Updates

Discussion

pic
Editor guide
Collapse
dariomarv profile image
DarioMarv

Hi Oz!
Thank you for sharing. I think that his web app is amazing.
I contact you to know if you never had the need to sync the true mouse pointer with the red pointer to maintain the hover effect inside a webpage.

All the best
Dario