Unity MR Part 12: Improve Scene

👀 Stumbled here on accident? Start with the introduction!

📚 The aim of this article is to enhance the visual and interactive elements of our scene. We will focus on several key improvements:

1. Graphics Enhancement: We'll enhance the graphics by configuring the Universal Render Pipeline (URP) assets.
2. Remove Debug Plane Material: The debug plane material, used for initial testing, will be removed for a cleaner look.
3. Door Shadow Effects: We'll set up the scene so that the door casts shadows on planes that will now be invisible, adding depth and realism.
4. Create a 'Behind the Door' Scene: We will develop a scene that is only revealed when the door is open, adding an element of surprise and exploration.

This article incorporates some advanced techniques that can significantly elevate your MR project. If you encounter any challenges or need further clarification, feel free to send us a message. Additionally, you can refer to the final project in our GitHub repository dedicated to this article series.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository

Light Estimation

ℹ️ Light estimation for MR devices like the Meta Quest 3 involves assessing the real-world lighting conditions and applying that information to the virtual environment to create a more immersive and realistic experience.

Regrettably, as indicated by Unity in their forums, this feature is not currently supported.

Instantiate the door facing the user

Currently, the door Prefab is instantiated with a consistent rotation. Let's modify the MRArticleSeriesController Script so that the door faces the user upon instantiation. To do this, locate the OnButtonPressedRightAsync method in the MRArticleSeriesController Script and update it as follows.

private async void OnButtonPressedRightAsync(InputAction.CallbackContext context)
{
    Debug.Log("MRArticleSeriesController -> OnButtonPressedRightAsync()");

    if (doorInstance != null) {
        Debug.Log("MRArticleSeriesController -> OnButtonPressedRightAsync(): Door already instantiated");

        return;
    }

    if (rayInteractor.TryGetCurrent3DRaycastHit(out RaycastHit hit))
    {
        Pose pose = new(hit.point, Quaternion.identity);
        Result<ARAnchor> result = await anchorManager.TryAddAnchorAsync(pose);

        result.TryGetResult(out ARAnchor anchor);

        if (anchor != null) {
            // Instantiate the door Prefab
            doorInstance = Instantiate(door, hit.point, Quaternion.identity);

            // Unity recommends parenting your content to the anchor.
            doorInstance.transform.parent = anchor.transform;

            // Make the door face the user after instantiating
            doorInstance.transform.LookAt(new Vector3(
                Camera.main.transform.position.x,
                doorInstance.transform.position.y,
                Camera.main.transform.position.z
            ));
        }
    }
}

As you can see, we have added a call to doorInstance.transform.LookAt in the script. After making this addition, save the file and return to the Unity Editor to apply these changes.

ℹ️ For more information on the LookAt function and how it works, you can refer to the Unity documentation. This resource provides detailed insights into how LookAt can be used to orient objects in your scene: Unity - Scripting API: Transform.LookAt.

Update AR Default Plane

In this step, we will modify the material of the AR Default Plane to make it transparent while still allowing it to receive shadows. This adjustment enables us to see the shadows cast by the door on our floor. Implementing this change enhances the realism of our MR experience, as it allows virtual objects like the door to interact more naturally with the environment.

Go to Window → Package Manager and install the package https://github.com/Cyanilux/URP_ShaderGraphCustomLighting.git with Install package from git URL:

Once you have completed the installation, close the Package Manager. Now, edit the AR Default Plane by double-clicking it in the Project window.

Replace the default Material with ShadowReceiver as seen in the above screenshot. That’s all for the AR Default Plane.

If you can’t find ShadowReceiver via search you can also drag and drop the material from Packages/com.cyanilux.shadergraph-customlighting/Examples/ShadowReceiver.mat.

Optimize Lightning

In this step, we will focus on optimizing the Directional Light in our scene. You have the option to adjust the values as shown in the upcoming screenshot, or alternatively, you can modify the settings according to your own preferences and the specific requirements

Optimize URP

We are currently set to the Balanced quality level for the Android build. Let's switch to using the High Fidelity quality setting by default, as illustrated in the upcoming screenshot. This change will enhance the visual quality of our application, offering a more detailed and immersive experience on Android devices.

Next, we must disable HDR in our URP asset. HDR is not supported in this context and will cause the application to crash if enabled. You can locate the URP asset at Assets/Settings/URP-HighFidelity.asset. The HDR option is available as the first option under the Quality section.

The High Fidelity settings utilize the Renderer Features Screen Space Ambient Occlusion, which significantly impacts performance. To optimize, you can either disable this feature or remove it entirely —both approaches are effective. You can find this setting in the asset located at Assets/Settings/URP-HighFidelity-Renderer.asset.

The behind the door

Let's create a straightforward effect that simulates a scene visible only when the door is open and the user looks through it. This approach will add an intriguing layer of interactivity to our project, enhancing the user experience by revealing a hidden scene.

Edit our door prefab by double-clicking Assets/Prefabs/Door in the Project window.

1. Disable the Door mesh through the inspector so we can look through the door.
2. Add a plane into the root of the prefab via Create → 3D → Plane.
3. Use the following Transform values as follows:

The prefab should now resemble the appearance displayed in the upcoming screenshot.

As you will notice, this method effectively creates the illusion where the "inside" of the door is only visible from the front. To observe this effect, take a look at the door from the back in the Scene view, as illustrated in the upcoming screenshot.

Now create a new Render Texture in Assets/Materials and name it DoorPlaneRenderTexture. Edit the Render Texture as follows:

As you can observe, we have set the Size to 1024x2048, which correlates with our plane's approximate dimensions of a 1:2 ratio. This size selection ensures that the texture or material applied to the plane is properly scaled and aligned, reflecting the plane's physical proportions in the virtual environment.

Now create a new Material in Assets/Materials and name it DoorPlaneRenderTexture. Edit the Material as follows:

Now, choose the DoorPlaneRenderTexture as the Base Map as seen in the next screenshot.

For the final step, assign the DoorPlaneRenderTexture material to the Mesh Renderer of the Plane.

Now, let's create a simple scene that will be rendered onto the plane. Before we start, make sure to return to the SampleScene and exit the prefab editor.

1. In your hierarchy, create an empty GameObject. You can do this by selecting Create Empty. Name this GameObject DoorPlaneScene.
2. Ensure the position of DoorPlaneScene is set to X: 0, Y: 0, Z: 0, unless it's already positioned there.
3. Create another empty GameObject as a child of DoorPlaneScene and name it CameraOffset . Set the Y position of CameraOffset to 1.1176 in its Transform properties.
4. To the CameraOffset GameObject, add a Camera component. This camera will capture the scene for rendering onto the plane.
5. Add another empty GameObject under DoorPlaneScene and name it Scene. This GameObject will hold the elements of your rendered scene.
6. Inside Scene, create a Cube (3D Object → Cube). Position this Cube with its Z coordinate set to 5.

These steps set up a basic scene structure, with a camera positioned to capture the scene, and a simple Cube as a visual element. The scene will be rendered from the perspective of the added camera.

The result looks as follows:

We aim for our new Camera to render only the contents of the Scene GameObject. To achieve this, navigate to Layers → Edit Layers..., as illustrated in the upcoming screenshot. Here, add a new user layer and name it DoorPlaneScene. This step is crucial for ensuring that the camera specifically captures the elements within the Scene, isolating its view to this particular part of your project for targeted rendering.

Now, select the DoorPlaneScene GameObject in the hierarchy and change its layer to DoorPlaneScene:

Confirm by also changing the layer for all children:

Lets configure the new Camera. Select our Main Camera in the hierarchy and copy the component values as seen in the next screenshot.

Now select the new Camera again and paste the component values:

Now, let's configure some additional settings for the new Camera:

1. Set the tag of the Camera to Untagged.
2. Under the Rendering tab, change the Culling Mask to include only the DoorPlaneScene and Ignore Raycast. This ensures the camera renders only the objects in the DoorPlaneScene layer.
3. Assign DoorPlaneRenderTexture as the Output Texture. This means the camera's view will be rendered to this texture.
4. Set the Far value under Clipping Planes to 250. This determines how far the camera can see.
5. Choose Skybox as the Background Type under the Environment tab.

After making these changes, your camera should be set up as shown in the following screenshot.

Finally, create a prefab from the DoorPlaneScene and then remove it from the hierarchy, following the same process we used for the Reticle in the Raycasts article.

Our plan is to incorporate the DoorPlaneScene into our Door prefab. This way, DoorPlaneScene will only be rendered when necessary. To do this, edit the Door prefab and drag and drop the DoorPlaneScene into the root of the prefab. This action is depicted in the upcoming screenshot and ensures that the scene is efficiently integrated with the Door prefab.

Save the prefab and return to the SampleScene.

Select the Main Camera again and set the Culling Mask from Everthing to: Default, TransparentFX, Ignore Raycast Water and UI.

Drag and drop the prefab Door into your scene to investigate the changes we made. The plane now should look as follows:

As you can observe, we now have our new scene projected onto the plane inside the door.

Next, re-enable the Door mesh, which we had disabled in a previous step. However, there's still an issue to address: the scene is static and doesn't adapt to the user's movements. Let's resolve this.

Create a new script and name it MimicCamera, then attach it to our newly added Camera. The script will be structured as follows:

using System.Collections;
using System.Collections.Generic;
using UnityEngine;

namespace Taikonauten.Unity.ArticleSeries
{
    public class MimicCamera : MonoBehaviour
    {
        public float offsetDistance = 30.0f;
        [SerializeField] private GameObject sceneCamera;
        private Camera mainCamera;
        public Transform DoorTransform { get; set; } = null;

        void Awake() {
            mainCamera = Camera.main;
        }

        void Update()
        {
            if (DoorTransform == null) {
                return;
            }

            // Calculate the direction and position offset from the door to the main camera
            Vector3 directionToCamera = mainCamera.transform.position - DoorTransform.position;
            Vector3 sceneCameraPosition = DoorTransform.position + directionToCamera.normalized * offsetDistance;

            sceneCameraPosition.y = sceneCameraPosition.y > 2f ? 2f : sceneCameraPosition.y;

            // Update the position and rotation of the scene camera
            sceneCamera.transform.position = sceneCameraPosition;
            sceneCamera.transform.rotation = Quaternion.LookRotation(DoorTransform.position - sceneCameraPosition);
        }
    }
}

Select the required values for the MimicCamera script as seen in the following screenshot:

We still need to assign the DoorTransform when the Door is instantiated. To do this, edit the MRArticleSeriesController script and add a private field for the MimicCamera script. This modification is shown on line 1 in the following code snippet.

Afterward, assign the DoorTransform following our LookAt method call. If you need guidance, you can also refer to the code in the 12_ImproveScene project for a clearer understanding 12_ImproveScene.

...

// Make the door face the user after instantiating
doorInstance.transform.LookAt(new Vector3(
    Camera.main.transform.position.x,
    doorInstance.transform.position.y,
    Camera.main.transform.position.z
));

doorInstance.GetComponentInChildren<MimicCamera>().DoorTransform = doorInstance.transform;

...

Adding a complex scene behind the door

For the final step, we'll upgrade from the simple cube to a more complex scene. In our example, we utilized the Free Low Poly Nature Forest asset, which you can find here: Free Low Poly Nature Forest. Add the asset to your project.

Find the 'Demo_01 scene from the package within the Assets/Pure Poly/Free Low Poly Nature Pack/Scenes directory. Then, drag and drop it into your hierarchy, as demonstrated in the upcoming screenshot.

If the Demo_01 scene looks as follows, we need to replace the shader that is used for the material.

Open the material located here: Assets/Pure Poly/Free Low Poly Nature Pack/Materials/PP_Standard_Material. Edit the material as follows:

1. Set the shader to Universal Render Pipeline/Lit.
2. As the Base Map choose the file PP_Color_Palette.
3. Set Smoothness to 0 for the Metallic Map.

Copy the Free_Forest GameObject, then edit our DoorPlaneScene prefab. Within the prefab, replace the cube with the Free_Forest GameObject, as illustrated in the next screenshot:

Set the Transform of the Free_Forest GameObject as follows:

Also ensure, that the layer of Free_Forest is set to DoorPlaneScene:

Save the prefab and return to your scene. Now remove the Demo_01 scene from your hierarchy:

This enhancement significantly elevates the visual appeal and immersion of the scene, demonstrating the versatility and potential of using varied assets in your MR environment.

ℹ️ If you find yourself facing any difficulties, remember that you can always refer to or download the code from our accompanying GitHub repository. 12_ImproveScene.

Testing the app

We are now ready to test the final version of our app. Here's what you need to do:

1. Choose Build and Run in Unity.
2. Once the app is running, press the trigger on the left controller.
3. You should see a label displaying the message "...Listening...".
4. Say the phrase “open the door”.
5. After a short delay, the animation of the door opening should begin.
6. Feel free to move around and explore what lies behind the door.

This test will allow you to experience the full functionality of the app, from voice command recognition to the dynamic opening of the door and the revealing of the scene behind it. Enjoy the immersive experience of exploring the new environment you've created!

👏