DEV Community πŸ‘©β€πŸ’»πŸ‘¨β€πŸ’»

Yuiko Ito
Yuiko Ito

Posted on • Updated on

React + TypeScript: Face detection with Tensorflow

Hello guys,

I have developed an application with face detection, that applies a mask automatically to your face in real time.

Alt Text

In this article, I will explain how to develop this application.

DEMO→https://mask-app-one.vercel.app/
github→https://github.com/YuikoIto/mask-app

This application has no loading animation, so you have to wait for some seconds at the first loading.

Setup react application and install react-webcam

$ npx create-react-app face-mask-app --template typescript
$ yarn add react-webcam @types/react-webcam
Enter fullscreen mode Exit fullscreen mode

Then, try setting up web camera.

// App.tsx

import { useRef } from "react";
import "./App.css";
import Webcam from "react-webcam";

const App = () => {
  const webcam = useRef<Webcam>(null);

  return (
    <div className="App">
      <header className="header">
        <div className="title">face mask App</div>
      </header>
      <Webcam
        audio={false}
        ref={webcam}
        style={{
          position: "absolute",
          margin: "auto",
          textAlign: "center",
          top: 100,
          left: 0,
          right: 0,
        }}
      />
    </div>
  );
}

export default App;
Enter fullscreen mode Exit fullscreen mode

yarn start and access http://localhost:3000/.

Alt Text

Yay! Web camera is now available.

Try Face detection using Tensorflow

Here, we are using this model. https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection

$ yarn add @tensorflow-models/face-landmarks-detection @tensorflow/tfjs-core @tensorflow/tfjs-converter @tensorflow/tfjs-backend-webgl
Enter fullscreen mode Exit fullscreen mode
  • If you don't use TypeScript, you don't have to install all of them. Install @tensorflow/tfjs instead of @tensorflow/tfjs-core, @tensorflow/tfjs-converter, and @tensorflow/tfjs-backend-webgl.

library versions

    "@tensorflow-models/face-landmarks-detection": "^0.0.3",
    "@tensorflow/tfjs-backend-webgl": "^3.6.0",
    "@tensorflow/tfjs-converter": "^3.6.0",
    "@tensorflow/tfjs-core": "^3.6.0",
Enter fullscreen mode Exit fullscreen mode
// App.tsx

import "@tensorflow/tfjs-core";
import "@tensorflow/tfjs-converter";
import "@tensorflow/tfjs-backend-webgl";
import * as faceLandmarksDetection from "@tensorflow-models/face-landmarks-detection";
import { MediaPipeFaceMesh } from "@tensorflow-models/face-landmarks-detection/dist/types";

const App = () => {
  const webcam = useRef<Webcam>(null);

  const runFaceDetect = async () => {
    const model = await faceLandmarksDetection.load(
      faceLandmarksDetection.SupportedPackages.mediapipeFacemesh
    );
  /*
     Please check your library version.
     The new version is a bit different from the previous.
     You should write as followings in the new one.
     You will see more information from https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection.
     const model = faceLandmarksDetection.SupportedModels.MediaPipeFaceMesh;
      const detectorConfig = {
        runtime: 'mediapipe', // or 'tfjs'
        solutionPath: 'https://cdn.jsdelivr.net/npm/@mediapipe/face_mesh',
      }
      const detector = await faceLandmarksDetection.createDetector(model, detectorConfig);
    */
    detect(model);
  };

  const detect = async (model: MediaPipeFaceMesh) => {
    if (webcam.current) {
      const webcamCurrent = webcam.current as any;
      // go next step only when the video is completely uploaded.
      if (webcamCurrent.video.readyState === 4) {
        const video = webcamCurrent.video;
        const predictions = await model.estimateFaces({
          input: video,
        });
        if (predictions.length) {
          console.log(predictions);
        }
      }
    };
  };

  useEffect(() => {
    runFaceDetect();
  // eslint-disable-next-line react-hooks/exhaustive-deps
  }, [webcam.current?.video?.readyState])

Enter fullscreen mode Exit fullscreen mode

Check logs.

image

OK, seems good.

Setup canvas to overlay the mask on your face

Add <canvas> under <Webcam>.

//App.tsx
const App = () => {
  const webcam = useRef<Webcam>(null);
  const canvas = useRef<HTMLCanvasElement>(null);

  return (
    <div className="App">
      <header className="header">
        <div className="title">face mask App</div>
      </header>
      <Webcam
        audio={false}
        ref={webcam}
      />
      <canvas
        ref={canvas}
      />

Enter fullscreen mode Exit fullscreen mode

Match the size of the canvas with the video.

    const videoWidth = webcamCurrent.video.videoWidth;
    const videoHeight = webcamCurrent.video.videoHeight;
    canvas.current.width = videoWidth;
    canvas.current.height = videoHeight;
Enter fullscreen mode Exit fullscreen mode

Then, let's see this map and check where we should fill out.

By this map, No. 195 is around the nose. So set this point as the fulcrum. Let's draw a mask easily by using beginPath()γ€œclosePath().

// mask.ts

import { AnnotatedPrediction } from "@tensorflow-models/face-landmarks-detection/dist/mediapipe-facemesh";
import {
  Coord2D,
  Coords3D,
} from "@tensorflow-models/face-landmarks-detection/dist/mediapipe-facemesh/util";

const drawMask = (
  ctx: CanvasRenderingContext2D,
  keypoints: Coords3D,
  distance: number
) => {
  const points = [
    93,
    132,
    58,
    172,
    136,
    150,
    149,
    176,
    148,
    152,
    377,
    400,
    378,
    379,
    365,
    397,
    288,
    361,
    323,
  ];

  ctx.moveTo(keypoints[195][0], keypoints[195][1]);
  for (let i = 0; i < points.length; i++) {
    if (i < points.length / 2) {
      ctx.lineTo(
        keypoints[points[i]][0] - distance,
        keypoints[points[i]][1] + distance
      );
    } else {
      ctx.lineTo(
        keypoints[points[i]][0] + distance,
        keypoints[points[i]][1] + distance
      );
    }
  }
};

export const draw = (
  predictions: AnnotatedPrediction[],
  ctx: CanvasRenderingContext2D,
  width: number,
  height: number
) => {
  if (predictions.length > 0) {
    predictions.forEach((prediction: AnnotatedPrediction) => {
      const keypoints = prediction.scaledMesh;
      const boundingBox = prediction.boundingBox;
      const bottomRight = boundingBox.bottomRight as Coord2D;
      const topLeft = boundingBox.topLeft as Coord2D;
      // make the drawing mask larger a bit
      const distance =
        Math.sqrt(
          Math.pow(bottomRight[0] - topLeft[0], 2) +
            Math.pow(topLeft[1] - topLeft[1], 2)
        ) * 0.02;
      ctx.clearRect(0, 0, width, height);
      ctx.fillStyle = "black";
      ctx.save();
      ctx.beginPath();
      drawMask(ctx, keypoints as Coords3D, distance);
      ctx.closePath();
      ctx.fill();
      ctx.restore();
    });
  }
};

Enter fullscreen mode Exit fullscreen mode

Import this draw function in App.tsx and use it.


    const ctx = canvas.current.getContext("2d") as CanvasRenderingContext2D;
    requestAnimationFrame(() => {
      draw(predictions, ctx, videoWidth, videoHeight);
    });
Enter fullscreen mode Exit fullscreen mode

Finish!

Thanks for reading.
This is my first time to use Tensorflow but thanks to a good README of the official github repository, I can make a small application easily. I will develop more with using Tensorflow 🐣

🍎🍎🍎🍎🍎🍎

Please send me a message if you need.

🍎🍎🍎🍎🍎🍎

References

Top comments (12)

Collapse
 
james_palermo_bc208e463e4 profile image
James Palermo

Fun project! 😎

Collapse
 
yuikoito profile image
Yuiko Ito

Thanks !

Collapse
 
vitalii0927 profile image
sapronov1021

Hi, @yuikoito

Does this work on mobile as a Selfie?

Collapse
 
yuikoito profile image
Yuiko Ito

Yes it works on my environment. (ios)

Collapse
 
vitalii0927 profile image
sapronov1021

Thanks, Yuiko

Thread Thread
 
vitalii0927 profile image
sapronov1021

@yuikoito , do you know how to get screenshot automatically, when doing selfie?

Thread Thread
 
yuikoito profile image
Yuiko Ito

@vitalii0927
sorry for my late reply.
I don't really know how to get screenshot "automatically" but maybe you can use "getScreenshot" method as follows.

  const webcamRef = useRef<Webcam>(null);
  const capture = useCallback(() => {
    const imageSrc = webcamRef.current?.getScreenshot();
    // write anything about imageSrc
  }, [webcamRef]);
Enter fullscreen mode Exit fullscreen mode

Then maybe you can loop this for example in requestAnimationFrame.

Thread Thread
 
vitalii0927 profile image
sapronov1021

Thank you, @yuikoito

Collapse
 
basicvisual profile image
basicvisual

great post. Have you tried similarily media pipe pose. I did try it and got misc resutls . But i didnt manage to get the 3d landmarks up and running

Collapse
 
yuikoito profile image
Yuiko Ito

Thank you for reading! No I didn't try other media pipe pose yet. I will try :)
But actually at first I tried to use face-api.js but I couldn't manage it. I searched how the face-api.js is running and which library is based on, and I found Tensorflow.
I am really beginner for Machine Learning and I still have many things I don't understand but I am happy if my article would help you!

Collapse
 
basicvisual profile image
basicvisual

I am the same here , Just learning by experimenting.

Thread Thread
 
yuikoito profile image
Yuiko Ito • Edited on

Hey I just posted a new one about TensorFlow.js. If you are interested, please have a look! This is not about 3d landmarks, but I hope my article help your learning :)

dev.to/yuikoito/tensorflow-next-js...

Let's Get Hacking

Join the DEV x Linode Hackathon 2022 and use your ingenuity and creativity to build using Linode.

β†’ Join the Hackathon <-