DEV Community

loading...

How to start and stop videos from a distance with your hand ✋

checkboxoz profile image Oz Ramos ・2 min read

In this tutorial you'll learn how to use BodyPix, through Handsfree.js, to start and stop a YouTube video by raising up your hand ✋

The Setup

Let's start by adding our dependencies:

<head>
  <!-- Handsfree dependencies -->
  <link rel="stylesheet" href="https://unpkg.com/handsfree@6.1.3/dist/handsfree.css" />
  <script src="https://unpkg.com/handsfree@6.1.3/dist/handsfree.js"></script>

  <!-- YouTube dependencies -->
  <script defer src="https://www.youtube.com/iframe_api"></script>
</head>
<body>
  <div id="player"></div>
</body>

We'll then use the YouTube iframe API to instantiate a video:

/**
  * This method is automatically called by the YouTube API
  * @see https://developers.google.com/youtube/iframe_api_reference
  */
let player
function onYouTubeIframeAPIReady () {
  player = new YT.Player('player', {
    // I picked this video because it's ad free
    videoId: 'AFKktZURpJ8'
  })
}

Setting up BodyPix through Handsfree.js

The next step is to instantiate Handsfree with the bodypix model enabled, and then create a plugin to watch for when the hand is raised. Handsfree.js plugins are simply callbacks that run after a model does inference.

// Create one instance for each camera you plan to use. By default it uses the main webcam (the selfie camera on mobile devices)
const handsfree = new Handsfree({
  models: {
    // The head model is enabled by default, so lets disable it
    head: {enabled: false},
    bodypix: {enabled: true}
  }
})


/**
  * Create a plugin that stops the video when the hand is up
  */
Handsfree.use('hand.stopVideo', {
  // Whether we have entered "hands have been up" state
  isBeingHeld: false,

  /**
   * This method is called once per webcam frame by Handsfree.js
   * @param {Object} The handsfree instance. Here we destructure it to just get instance.body
   */
  onFrame ({body}) {
    // only continue if one of the wrists are visible
    if (!body.pose || !body.pose.keypoints
      || (body.leftWrist.score < 0.6 && body.rightWrist.score < 0.6)) return

    // Check if one wrist is level with that shoulder
    if (body.leftWrist.y <= body.leftShoulder.y || body.rightWrist.y <= body.rightShoulder.y) {
      if (!this.isBeingHeld) {
        this.toggleVideo()
        this.isBeingHeld = true
      }
    } else {
      this.isBeingHeld = false
    }
  },

  /**
    * Toggle the video
    */
  toggleVideo () {
    const state = player.getPlayerState()

    if (state === 1) {
      player.pauseVideo()
    } else {
      player.playVideo()
    }
  }
})

That's all there is too it!

This tutorial quickly covered how to instantiate a YouTube video using the YouTube Iframe API. We then learned how to instantiate Handsfree with the bodypix model, and check for when the hand is raised...but there's so much more you can do!

Here are some other links you might find useful:

Thanks for reading and happy coding 👋

Discussion

pic
Editor guide