Augmented Faces with Unity and ARCore

Last time we created an Augmented Reality app that recognized vertical planes (like walls) and added a virtual painting there. You can check the tutorial from here if you haven't seen it.

This time we will create face filters! The face filter takes inspiration from one of my favorite videogames Horizon Zero Dawn. The finished project is in my Github if you just want to test this out without coding it yourself.

This tutorial assumes you have at least basic understanding of how Unity works (what are GameObjects and Prefabs etc) so if you are new to Unity I recommend you to first go through their beginner tutorial Roll-a-Ball that explains the basics well!

Start by downloading a base project from my Github or follow the first tutorial from this series before going forward.

Adding face detection

Open the Unity project and click on AR Session Origin from Hierarchy and click on AR Camera. Switch the camera's facing direction to User.

Click on AR Session Origin and Add Component. Search for AR Face Manager script and add it.

Then right click on Hierarchy and choose XR -> AR Default Face. Create a folder called Prefabs on Assets and drag and drop AR Default Face there and then delete it from Hierarchy.

Drag the created prefab from Assets to AR Face Manager's Face Prefab (in AR Session Origin).

If you Build and Run the app now, you will see a bit weird looking green face augmented on your detected face.

Adding hologram

Download a hologram 3D model from here. It takes inspiration from one of my favorite video games Horizon Zero Dawn, where the main character Aloy uses a device which created a similar kind of hologram.

Download everything from the hologram folder (link above) and create a new folder in your project's Assets and copy them there. There should be a HologramPrefab which already has all the needed settings and components. It's Position should be X = -0.09, Y = 0.05 and Z = 0.02, Rotation X = 0, Y = -115 and Z = 0 and Scale X, Y and Z 1.2.

It also has a ControllingGlitch script and Hologram Material. I followed a tutorial made by sharpcoderblog.com so check their tutorial out if you want to understand better how the Hologram shader and material works.

Open the AR Default Face Prefab that we created earlier by clicking it once and then clicking a button Open Prefab.

Add HologramPrefab as a child of AR Default Face. Make sure it's position, rotation and scale stays the same.

We still need to change the material of the parent GameObject (AR Default Face). Currently it is still using the green default material. Right click on Assets folder and click Create -> Shader -> Unlit Shader. This creates a new shader file. Name it FaceOccluder, open it and copy paste the code below:

// Tutorial for shader from https://tutorialsforar.com/how-to-hide-objects-using-ar-foundation-in-unity/
Shader "Custom/FaceOccluder"
{
    SubShader
    {
        Tags { "RenderType"="Opaque" }
        Tags { "Queue" = "Geometry-1" }
        ZWrite On
        ZTest LEqual
        ColorMask 0

        Pass
        {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
            };

            struct v2f
            {
                float4 vertex : SV_POSITION;
            };

            v2f vert (appdata v)
            {
                v2f o;
                o.vertex = UnityObjectToClipPos(v.vertex);
                return o;
            }

            fixed4 frag (v2f i) : SV_Target
            {
                return fixed4(0.0, 0.0, 0.0, 0.0);
            }
            ENDCG
        }
    }
}

I'm not going into details how shaders work, because it's a big subject, but basically it calculates each pixel and doesn't draw the ones that are located around our face. This way the hologram will disappear behind our face when rotating our head enough. It makes the hologram feel more real. You can read more in the tutorial where I found the occlusion shader code: tutorialsforar.com.

Next right click on Assets folder and choose Create -> Material and name it FaceOccluder. Drag and drop the shader file we created earlier on the material (or click on the Material and choose the Shader manually like in the picture above).

Then drag and drop the material on AR Default Face.

Finished

Now everything's done! You can build the project and you should see a hologram following your head movement!

Give me a follow if you want to see more tutorials! You can also follow my Instagram whatminjaplays if you are interested to see more about my days as a software developer and a gaming enthusiast!