Augmented Images using ARCore and Unity

Last time we installed Unity game engine and configured ARCore plugin. You can check the tutorial from here if you haven't seen it.

This time we will create an Android AR app that displays 3D models on images! The finished project is in my Github if you just want to test this out without coding it yourself.

This tutorial assumes you have at least basic understanding of how Unity works (what are GameObjects and Prefabs etc) so if you are new to Unity I recommend you to first go through their beginner tutorial Roll-a-Ball that explains the basics well!

Augmented Images

Augmented Images API in ARCore let's you to use computer vision algorithm to display 3D models on images in your environment, like on a postcard, a poster or a product package.

There are different SDKs for building applications like this (for example Vuforia) but on this series I'm only using Google ARCore.

Augmented Images API is able to display up to 20 pictures simultaneously. Check the best practices from ARCore documentation for information about what kind of images are good for being tracked.

The image I'm using is not the best but it works well enough. The 3D model might be a little unstable and you might see it shaking a bit but I'm using these on this tutorial because I have made them both myself so there won't be any copyright issues!

Getting started

Create a new Unity project and follow the instructions on my previous tutorial or download the finished project template from my Github and open it.

Click on ARSession Origin on Unity Hierarchy and Add Component. Search for AR Tracked Image Manager and add it.

Right click on Assets -> Create -> XR -> Reference Image Library. You can use your own photo or download the one I used from here.

I'm using a Frog 3D model that I have made and an image of it in this project. Click on the created Reference Image Library and Add Image. Select the image of a frog (or whatever image you want to use).

Give the image a name and you can specify the size of the (physical) image if you know it but it's not necessary.

Add the reference to a created Reference Image Library to the AR Session Origin.

Creating the prefab that gets instantiated

Next we can add the prefab that will get instantiated when the image added to the Reference Image Library gets detected.

Right click on the Hierarchy, choose Create Empty and name it ARModel. Download the frog 3D model from here and add it to the Unity's Asset folder. Drag and drop the frog model to the hierarchy and move it as a child of ARModel. Make sure that the child frog model is positioned in 0, 0, 0.

You can reset the position of a model by clicking the three dots in the Transform and choosing Reset. Change the Scale of the frog model to be 0.05 on all (X, Y, Z) axes or it will be huge otherwise. Also rotate the model -90 degrees on X axis and 180 degrees on Y Axis. This is because we are scanning an image on the screen, so the frog will be positioned correctly for that. If you are using a printed image and want the frog to stand on top of it, keep the frog's X, Y and Z rotation on 0.

Drag and drop the created ARModel to a Prefabs folder and delete ARModel from the Hierarchy.

Then add a reference to ARModel to Tracked Image Prefab that's in AR Session Origin.

Finished

Now everything's done! You can build the project and you should see a frog model instantiated when you point the camera to the frog image! If the 3D model doesn't jump on top of the image, try rotating the phone a bit around the image. Also try changing the brightness of your monitor. The example image is not the best quality and there might be reflections on the screen which might make the image even harder to detect, but it should still work!

Give me a follow if you want to see more tutorials! You can also follow my Instagram whatminjaplays if you are interested to see more about my days as a software developer and a gaming enthusiast!