Plane Detection with ARCore and Unity

Last time we went through how to detect images and display 3D content on top of it. You can check the tutorial from here if you haven't seen it. This time we use Plane detection to recognize walls and add a 3D model of a painting there.

The finished project is in my Github if you just want to test this out without building it yourself.

This tutorial assumes you have at least basic understanding of how Unity works (what are GameObjects and Prefabs etc) so if you are new to Unity I recommend you to first go through their beginner tutorial Roll-a-Ball that explains the basics well!

What is Plane Detection

ARCore is constantly trying to understand the environment where it's used. It tries to detect feature points and planes, such as walls, floors or tables. However, flat surfaces that don't have any texture, such as white walls, might have some difficulties to be detected because of lack of feature points. There's a better Depth API that can be used for understanding the environment better, but because this is a beginner tutorial I decided to use Plane Detection.

Adding Plane Detection to Unity project

Follow the first tutorial from my AR intro series to setup a new Unity project for Augmented Reality use or download the base project from here.

Choose AR Session Origin from Hierarchy and add AR Plane Manager to it. Set the Detection Mode to Vertical so we are able to detect walls.

Then right click on Hierarchy and choose XR -> AR Default Plane. Create a new folder called Prefabs on Assets and drag and drop AR Default Plane there and then delete it from the Hierarchy. Drag and drop the created prefab to AR Plane Manager.

Creating a Picture Frame

Download a picture frame 3D model from here and an image of a low poly world from here. Put them in the project's Asset folder.

Right click on Hierarchy and choose Create empty. Name the created GameObject PictureFrame and make sure it's Position and Rotation is at 0, 0, 0 and that it's Scale is 1, 1, 1.

Drag and drop picture-frame.fbx 3D model to Hierarchy and put it as a child of PictureFrame. Change it's rotation to be X 0, Y 180, Z 0 and scale to X 20, Y 25, Z 20.

Right click on Assets and choose Create -> Material. Name it PictureFrameMaterial and choose a color you want it to be. I used brown #9F5215 HEX color. Drag and drop the material on picture-frame 3D model.

Click on the low-poly-world image on Assets folder. Change the Texture Type to be Sprite and Pixels Per Unit to 500. Drag and drop the image to the Hierarchy and put it as a child of PictureFrame. Change it's scale to be X 0.6, Y 0.6 and Z 0.6.

Click on PictureFrame (parent) GameObject and add a new tag Spawnable to it.

Drag and drop PictureFrame GameObject to Prefab folder and delete it from the Hierarchy.

Spawning the object

Next right click on Assets and choose Create -> C# Script. Name it SpawnableManager.

Open it and add the following code:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.XR.ARFoundation;
using UnityEngine.UI;

public class SpawnableManager : MonoBehaviour
{
    [SerializeField] private ARRaycastManager _raycastManager;
    [SerializeField] private GameObject _spawnablePrefab;

    private List<ARRaycastHit> _raycastHits = new List<ARRaycastHit>();
    private GameObject _spawnedObject;

    private void Start()
    {
        _spawnedObject = null;
    }

    private void Update()
    {
        // No touch events
        if (Input.touchCount == 0)
        {
            return;
        }

        // Save the found touch event
        Touch touch = Input.GetTouch(0);

        if (_raycastManager.Raycast(touch.position, _raycastHits))
        {
            // Beginning of the touch, this triggers when the finger first touches the screen
            if (touch.phase == TouchPhase.Began)
            {
                // Spawn a GameObject
                SpawnPrefab(_raycastHits[0].pose.position);
            }
            // Finger still touching the screen and moving, GameObject has already been instantiated
            else if (touch.phase == TouchPhase.Moved && _spawnedObject != null)
            {
                // Moves the spawned GameObject where the finger is moving
                _spawnedObject.transform.position = _raycastHits[0].pose.position;
            }

            // Finger lifted from the screen
            if (touch.phase == TouchPhase.Ended)
            {
                // Don't track the object anymore
                _spawnedObject = null;
            }
        }
    }


    // Instantiate a GameObject to the location where finger was touching the screen
    private void SpawnPrefab(Vector3 spawnPosition)
    {
        _spawnedObject = Instantiate(_spawnablePrefab, spawnPosition, Quaternion.identity);
    }
}

This code detects if the screen is touched. It then sends a Raycast to detect collision with AR Plane, which is created when AR Plane Manager detects vertical planes (walls).

Save the script and go back to Unity Editor. Click on AR Session Origin, click on Add Component and search for AR Raycast Manager and add it. Then right click on Hierarchy and Create Empty. Name it SpawnableManager and drag and drop the SpawnableManager script we just created on it.

Drag and drop AR Session Origin to Raycast Manager and PictureFrame prefab to Spawnable Prefab.

Adding UI

Now we have the functionality to detect vertical planes (walls) and we can spawn a picture frame to it. Next let's add a button which disables the planes so we are only seeing the picture frame.

Right click on Hierarchy and add UI -> Button and name it HidePlanesButton. Make its Width and Height 200.

Click the Anchor Presets button (red circle on the image above) and then press Ctrl + Alt and left click on blue circle. This makes the UI element to anchor itself to the upper left corner. Change the Pos X to 100 and Pos Y to -100.

Next create a new script and name it PlaneDetectionController and add the code below.

using System.Collections.Generic;
using UnityEngine;
using UnityEngine.UI;
using UnityEngine.XR.ARFoundation;

namespace UnityEngine.XR.ARFoundation.Samples
{
    /// <summary>
    /// This example demonstrates how to toggle plane detection,
    /// and also hide or show the existing planes.
    /// </summary>
    [RequireComponent(typeof(ARPlaneManager))]
    public class PlaneDetectionController : MonoBehaviour
    {
        [Tooltip("The UI Text element used to display plane detection messages.")]
        [SerializeField]
        Text m_TogglePlaneDetectionText;

        /// <summary>
        /// The UI Text element used to display plane detection messages.
        /// </summary>
        public Text togglePlaneDetectionText
        {
            get { return m_TogglePlaneDetectionText; }
            set { m_TogglePlaneDetectionText = value; }
        }

        /// <summary>
        /// Toggles plane detection and the visualization of the planes.
        /// </summary>
        public void TogglePlaneDetection()
        {
            m_ARPlaneManager.enabled = !m_ARPlaneManager.enabled;

            string planeDetectionMessage = "";
            if (m_ARPlaneManager.enabled)
            {
                planeDetectionMessage = "Disable Plane Detection and Hide Existing";
                SetAllPlanesActive(true);
            }
            else
            {
                planeDetectionMessage = "Enable Plane Detection and Show Existing";
                SetAllPlanesActive(false);
            }

            if (togglePlaneDetectionText != null)
                togglePlaneDetectionText.text = planeDetectionMessage;
        }

        /// <summary>
        /// Iterates over all the existing planes and activates
        /// or deactivates their <c>GameObject</c>s'.
        /// </summary>
        /// <param name="value">Each planes' GameObject is SetActive with this value.</param>
        void SetAllPlanesActive(bool value)
        {
            foreach (var plane in m_ARPlaneManager.trackables)
                plane.gameObject.SetActive(value);
        }

        void Awake()
        {
            m_ARPlaneManager = GetComponent<ARPlaneManager>();
        }

        ARPlaneManager m_ARPlaneManager;
    }
}

This code is Unity's sample code which toggles between enabling and disabling the plane detection. You can check all their samples from here. Save the script and go back to Unity editor.

Add the script to AR Session Origin and drag and drop the Text (underlined with green) object to Toggle Plane Detection Text.

Click on HidePlanesButton and add On Click() event. Click on the + and add AR Session Origin there. Then you can choose PlaneDetectionController -> TogglePlane Detection from the dropdown menu.

Now we only need to make sure we disable AR Raycast when we click on UI Button. Add a new script to Assets called Vector2Extensions and open it. Add the code below:

using UnityEngine;
using System.Collections.Generic;
using UnityEngine.EventSystems;

// Extension made by dilmerv
// - https://github.com/dilmerv/UnityARBlockARRaycast/
public static class Vector2Extensions
{
    public static bool IsPointOverUIObject(this Vector2 pos)
    {
        if (EventSystem.current.IsPointerOverGameObject())
        {
            return false;
        }

        PointerEventData eventPosition = new PointerEventData(EventSystem.current);
        eventPosition.position = new Vector2(pos.x, pos.y);

        List<RaycastResult> results = new List<RaycastResult>();
        EventSystem.current.RaycastAll(eventPosition, results);

        return results.Count > 0;
    }
}

This extension function checks if we have clicked an UI element and disables the AR Raycast if we have. I found this extension from dilmerv GitHub so make sure to credit them if you use their code!

Next open SpawnableManager and modify the TouchPhase.Began part:

...
if (_raycastManager.Raycast(touch.position, _raycastHits))
        {
            // Beginning of the touch, this triggers when the finger first touches the screen
            if (touch.phase == TouchPhase.Began)
            {
                bool isTouchOverUI = touch.position.IsPointOverUIObject();

                if (!isTouchOverUI)
                {
                    // Spawn a GameObject
                    SpawnPrefab(_raycastHits[0].pose.position);
                }
            }
...

This way we only instantiate the picture frame if we are not touching any UI elements.

Finished

Now we are finished! Build the project to your phone and try scanning a wall. You should see a plane being instantiated on it. If you then tap on the wall, a picture frame will appear. Click on the button to disable the plane detection and you should only see a picture frame on your wall!

Give me a follow if you want to see more tutorials! You can also follow my Instagram whatminjaplays if you are interested to see more about my days as a software developer and a gaming enthusiast!