DEV Community

kurohuku
kurohuku

Posted on • Edited on

SteamVR Overlay with Unity: Overlay Events

#unity3d #steamvr #openvr #vr

Show the current time on the right hand

Let’s display the current time on the right hand.

Create variable

Add a variable in WatchOverlay.cs to determine which hand to show the current time.
OpenVR has ETrackedControllerRole type representing the left or right hand, so we use this.

public class WatchOverlay : MonoBehaviour
{
    public Camera camera;
    public RenderTexture renderTexture;
+   public ETrackedControllerRole targetHand = EtrackedControllerRole.LeftHand;

    private ulong overlayHandle = OpenVR.k_ulOverlayHandleInvalid;

    ...
Enter fullscreen mode Exit fullscreen mode

The left hand is EtrackedControllerRole.LeftHand, and the right hand is EtrackedControllerRole.RightHand.

Get selected controller device index

Currently, it gets only the left controller device index. Let’s make it to get the current selected controller.

private void Update()
{
    var position = new Vector3(x, y, z);
    var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);

-   var leftControllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(ETrackedControllerRole.LeftHand);
-   if (leftControllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
-   {
-       Overlay.SetOverlayTransformRelative(overlayHandle, leftControllerIndex, position, rotation);
-   }
+   var controllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(targetHand);
+   if (controllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
+   {
+       Overlay.SetOverlayTransformRelative(overlayHandle, controllerIndex, position, rotation);
+   }

    Overlay.SetOverlayRenderTexture(overlayHandle, renderTexture);
}
Enter fullscreen mode Exit fullscreen mode

Run the program, switch Target Hand to Right Hand in the WatchOverlay inspector. It should display the current time on the right hand.

Image description

But the current time is displayed on the wrong position on the right hand because we set the position for only the left hand.

Image description

Let’s fix the position for the right hand.

Note down current params

At first, open WatchOverlay inspector, note down X, Y, Z, RotationX, Rotation Y, Rotation Z values.

Left hand params
x = -0.044
y = 0.015
z = -0.131
rotationX = 154
rotationY = 262
rotationZ = 0
Enter fullscreen mode Exit fullscreen mode

Create variables for each hand

Remove the member variables below.

  • x, y, z
  • rotationX, rotationY, rotationZ

Keep the size because it will be shared from both hands.

Add left-hand and right-hand variables below.

  • leftX, leftY, leftZ
  • leftRotationX, leftRotationY, leftRotationZ
  • rightX, rightY, rightZ
  • rightRotationX, rightRotationY, rightRotationZ 
public class WatchOverlay : MonoBehaviour
{
    public Camera camera;
    public RenderTexture renderTexture;
    public ETrackedControllerRole targetHand = ETrackedControllerRole.RightHand;

    private ulong overlayHandle = OpenVR.k_ulOverlayHandleInvalid;

    // Keep size
    [Range(0, 0.5f)] public float size;

-   // Remove current variables
-   [Range(-0.2f, 0.2f)] public float x;
-   [Range(-0.2f, 0.2f)] public float y;
-   [Range(-0.2f, 0.2f)] public float z;
-   [Range(0, 360)] public int rotationX;
-   [Range(0, 360)] public int rotationY;
-   [Range(0, 360)] public int rotationZ;

+   // Add for left hand
+   [Range(-0.2f, 0.2f)] public float leftX;
+   [Range(-0.2f, 0.2f)] public float leftY;
+   [Range(-0.2f, 0.2f)] public float leftZ;
+   [Range(0, 360)] public int leftRotationX;
+   [Range(0, 360)] public int leftRotationY;
+   [Range(0, 360)] public int leftRotationZ;

+   // Add for right hand
+   [Range(-0.2f, 0.2f)] public float rightX;
+   [Range(-0.2f, 0.2f)] public float rightY;
+   [Range(-0.2f, 0.2f)] public float rightZ;
+   [Range(0, 360)] public int rightRotationX;
+   [Range(0, 360)] public int rightRotationY;
+   [Range(0, 360)] public int rightRotationZ;
Enter fullscreen mode Exit fullscreen mode

Switch position and rotation to match the current hand

Edit Update() to set corresponding to targetHandposition and rotation.

private void Update()
{
-   var position = new Vector3(x, y, z);
-   var rotation = Quaternion.Euler(rotationX, rotationY, rotationZ);
+   Vector3 position;
+   Quaternion rotation;
+       
+   if (targetHand == ETrackedControllerRole.LeftHand)
+   {
+       position = new Vector3(leftX, leftY, leftZ);
+       rotation = Quaternion.Euler(leftRotationX, leftRotationY, leftRotationZ);
+   }
+   else
+   {
+       position = new Vector3(rightX, rightY, rightZ);
+       rotation = Quaternion.Euler(rightRotationX, rightRotationY, rightRotationZ);
+   }

    var controllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(targetHand);
    if (controllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
    {
        Overlay.SetOverlayTransformRelative(overlayHandle, controllerIndex, position, rotation);
    }

    Overlay.SetOverlayRenderTexture(overlayHandle, renderTexture);
}
Enter fullscreen mode Exit fullscreen mode

Adjust position for right hand

Run the program and open WatchOverlay inspector.
Check Target Hand is set to Right Hand.

Move the right-hand sliders to adjust the position and rotation to the best fit.

Image description

Here are the sample values for the right hand.

Right hand params
x = 0.04
y = 0.003
z = -0.107
rotationX = 24
rotationY = 258
rotationZ = 179
Enter fullscreen mode Exit fullscreen mode

Image description

After you set the best position on the sliders, right click theWatchOverlay component name on the inspector > Copy Component to copy values.

Image description

Stop the program, right click > Paste Component Values.
Also, input the left-hand values from the note you took.
Reset Target Hand variable to Left Hand.

Image description

Here, we got the position and rotation for both hands.

Create button events

Let’s make it so that when the button is clicked, switch controllers.
Create a new script WatchSettingController.cs inside Scripts folder.
We will add the code of button events here.

Right click hierarchy > Create Empty to make an empty object, and change the object name to SettingController.
Add WatchSettingController.cs to the SettingController object.

Image description

Copy the code below to WatchSettingController.cs.

using UnityEngine;

public class WatchSettingController : MonoBehaviour
{
    [SerializeField] private WatchOverlay watchOverlay;
}
Enter fullscreen mode Exit fullscreen mode

Add an event when the button is clicked to WatchSettingController.cs.

using UnityEngine;
+ using Valve.VR;

public class WatchSettingController : MonoBehaviour
{
    [SerializeField] private WatchOverlay watchOverlay;

+   public void OnLeftHandButtonClick()
+   {
+       // When "Left Hand" button is clicked, switch to left hand.
+       watchOverlay.targetHand = ETrackedControllerRole.LeftHand;
+   }
+
+   public void OnRightHandButtonClick()
+   {
+       // When "Right Hand" button is clicked, switch to right hand.
+       watchOverlay.targetHand = ETrackedControllerRole.RightHand;
+   }
}
Enter fullscreen mode Exit fullscreen mode

Set button events

Select Dashboard > Canvas > LeftHandButton object on the hierarchy.
On the inspector, click + button in the OnClick() field of Button component.
Drag SettingController object to None (Object) of OnClick() field, and select WatchSettingController.OnLeftHandButtonClick().

Image description

Here, SettingController.OnLeftHandButtonClick() is called when the button is clicked.

Similarly, set WatchSettingController.OnRightHandButtonClick() to OnClick() field in RightHandButton inspector.

Image description

Get dashboard event

Currently, nothing happens when the button is clicked.
The previous settings are for Unity side events, not OpenVR.

OpenVR and Unity events are different, so we must notify Unity of the OpenVR dashboard event.

First, we get the OpenVR click event.
Overlay events can be detected by polling with PollNextOverlayEvent(). (read the wiki for details)

Let’s watch dashboard overlay events in Update() with PollNextOverlayEvents().
If some overlay events occur on the specified overlay, PollNextOverlayEvent() returns true and takes one event from the event queue. If all events are taken, it returns false.

Add event detection code into Update() of DashboardOverlay.cs.

void Update()
{
    Overlay.SetOverlayRenderTexture(dashboardHandle, renderTexture);

+   var vrEvent = new VREvent_t();
+   var uncbVREvent = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(VREvent_t));
+
+   // If overlay events are left then true.
+   while (OpenVR.Overlay.PollNextOverlayEvent(dashboardHandle, ref vrEvent, uncbVREvent))
+   {
+       // vrEvent is popped up event.
+   }
+
+   // Break the loop when all events are popped out.
}
Enter fullscreen mode Exit fullscreen mode

OpenVR event represents as VREvent_t type. (read the wiki for details)
uncbVREvent is the byte size of VREvent_t type.

Get EVREventType.VREvent_MouseButtonDown and EVREventType.VREvent_MouseButtonUp events.

private void Update()
{
    var vrEvent = new VREvent_t();
    var uncbVREvent = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(VREvent_t));

    while (OpenVR.Overlay.PollNextOverlayEvent(overlayHandle, ref vrEvent, uncbVREvent))
    {
+       switch ((EVREventType)vrEvent.eventType)
+       {
+           case EVREventType.VREvent_MouseButtonDown:
+               Debug.Log("MouseDown");
+               break;
+
+           case EVREventType.VREvent_MouseButtonUp:
+               Debug.Log("MouseUp");
+               break;
+       }
    }
}
Enter fullscreen mode Exit fullscreen mode

vrEvent.eventType is a uint event code, so we cast it EVREventType for comparison.
Run the program, click the dashboard button in VR, and then “MouseDown” and “MouseUp” should be logged.

Image description

Image description
Aim with the laser pointer then click!

Is there EVREventType.VREvent_MouseClick?

No there is.
OpenVR Overlay mouse events are simple. There are only three events MouseButtonDown, MouseButtonUp, MouseMove. If you want to use other events like OnClick, OnMouseEnter or OnMouseLeave, you should combine the three basic events. It is harder than Unity UI to make some UIs like slider.

Is there easy way to make UI?

I released a UI asset that has basic UIs for SteamVR dashboard. If you are interested, try it.
https://assetstore.unity.com/packages/tools/gui/ovrle-ui-dashboard-ui-kit-for-steamvr-270636

OVRLE UI - Dashboard UI Kit for SteamVR - YouTube

Asset Store Pagehttps://u3d.as/3bHTOVRLE UI is an asset to construct a dashboard overlay. You can use this assset as an overlay application template out of t...

favicon youtube.com

Get the click position from the overlay mouse event.

private void Update()
{
    var vrEvent = new VREvent_t();
    var uncbVREvent = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(VREvent_t));

    while (OpenVR.Overlay.PollNextOverlayEvent(overlayHandle, ref vrEvent, uncbVREvent))
    {
        switch (vrEvent.eventType)
        {
            case (uint)EVREventType.VREvent_MouseButtonDown:
-               Debug.Log("MouseDown");            
+               Debug.Log($"MouseDown: ({vrEvent.data.mouse.x}, {vrEvent.data.mouse.y})");
                break;

            case (uint)EVREventType.VREvent_MouseButtonUp:
-               Debug.Log("MouseUp");
+               Debug.Log($"MouseUp: ({vrEvent.data.mouse.x}, {vrEvent.data.mouse.y})");
                break;
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

vrEvent has a mouse position as VREvent_Mouse_t type.
The mouse position is UV, so the x and y are 0–1.

Run the program, open the SteamVR dashboard, and click the overlay.
The clicked mouse position should be logged.

Image description

Apply mouse scaling factor

The mouse position is 0–1 UV, but we can convert it to the actual UI position (px) by applying the Mouse Scaling Factor with SetOverlayMouseScale(). (read the wiki for details)

Set mouse scaling factor in Start() of DashboardOverlay.cs.

private void Start()
{
    OpenVRUtil.System.InitOpenVR();

    (dashboardHandle, thumbnailHandle) = Overlay.CreateDashboardOverlay("WatchDashboardKey", "Watch Setting");

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    Overlay.SetOverlayFromFile(thumbnailHandle, filePath);

    Overlay.FlipOverlayVertical(dashboardHandle);
    Overlay.SetOverlaySize(dashboardHandle, 2.5f);

+   var mouseScalingFactor = new HmdVector2_t()
+   {
+       v0 = renderTexture.width,
+       v1 = renderTexture.height
+   };
+   error = OpenVR.Overlay.SetOverlayMouseScale(dashboardHandle, ref mouseScalingFactor);
+   if (error != EVROverlayError.None)
+   {
+       throw new Exception("Failed to set mouse scaling factor: " + error);
+   }
}
Enter fullscreen mode Exit fullscreen mode

v0 is width, v1 is height of mouseScalingFactor.
When the OpenVR mouse event is dispatched, the mouse scaling factor multiplies to the UV position to scale it to the actual UI size.

Run the program, and click the dashboard.
The click position should be scaled to (0, 0) ~ (1024, 768).

Image description

Test mouse event with Overlay Viewer

FYI, Overlay Viewer can dispatch mouse events.

Run the program, launch Overlay Viewer, and click WatchDashboardKey from the overlay list.
Check the right bottom Mouse Capture and click the preview area, A mouse event will be dispatched and logged to the Unity console.

Image description

It is useful to check events without the HMD.

Detect which button is clicked

Detect which Unity button is clicked by the event mouse position.
Unity’s Graphic Raycaster can detect UI components by specific mouse positions.

Get Graphic Raycaster

Add graphicRaycaster variable to DashboardOverlay.cs.

using UnityEngine;
using Valve.VR;
using System;
using OpenVRUtil;
+ using UnityEngine.UI;

public class DashboardOverlay : MonoBehaviour
{
    public Camera camera;
    public RenderTexture renderTexture;
+   public GraphicRaycaster graphicRaycaster;
    private ulong dashboardHandle = OpenVR.k_ulOverlayHandleInvalid;
    private ulong thumbnailHandle = OpenVR.k_ulOverlayHandleInvalid;

    ...
Enter fullscreen mode Exit fullscreen mode

Open Dashboard > DashboardOverlay inspector.
Drag Dashboard > Canvas object to the GraphicRaycaster variable.

Image description

Add detection method

Add a method to DashboardOverlay.cs which detects a UI component at the specified position.

private void OnDestroy()
{
    OpenVRUtil.System.ShutdownOpenVR();
}

+ private Button GetButtonByPosition(Vector2 position)
+ {
+    // Return button that at position.x, position.y.
+    // If nothing, return null.
+    return null;
+ }

...
Enter fullscreen mode Exit fullscreen mode

In this tutorial, we only use Button component so we take Button only.

Add EventSystem

GraphicRaycaster requires the PointerEventData as an argument. PointerEvent data is defined in EventSystem which controls Unity events.
At first, add a variable to save EventSystem to DashboardOverlay.cs.

using UnityEngine;
using Valve.VR;
using System;
using OpenVRUtil;
using UnityEngine.UI;
+ using UnityEngine.EventSystems;

public class DashboardOverlay : MonoBehaviour
{
    public Camera camera;
    public RenderTexture renderTexture;
    public GraphicRaycaster graphicRaycaster;
+   public EventSystem eventSystem;

    private ulong dashboardHandle = OpenVR.k_ulOverlayHandleInvalid;
    private ulong thumbnailHandle = OpenVR.k_ulOverlayHandleInvalid;

    ...
Enter fullscreen mode Exit fullscreen mode

Open Dashboard > DashboardOverlay inspector from the hierarchy.
Drag EventSystem object to the Event System variable.

Image description

Create PointerEventData

In GetButtonByPosition(), create PointerEventData with EventSystem.

private Button GetButtonByPosition(Vector2 position)
{
+   var pointerEventData = new PointerEventData(eventSystem);
+   pointerEventData.position = position;
    return null;
}
Enter fullscreen mode Exit fullscreen mode

Get Button with GraphicRaycaster

Pass the pointerEventData to GraphicRaycaster.Raycast() to find a Button component on the click position, and return it if it exists.

using UnityEngine;
using Valve.VR;
using System;
using OpenVRUtil;
using UnityEngine.UI;
using UnityEngine.EventSystems;
+ using System.Collections.Generic;

public class DashboardOverlay : MonoBehaviour
{
    public Camera camera;
    public RenderTexture renderTexture;
    public GraphicRaycaster graphicRaycaster;
    public EventSystem eventSystem;

    ...

    private Button GetButtonByPosition(Vector2 position)
    {
        var pointerEventData = new PointerEventData(eventSystem);
        pointerEventData.position = position;

+       // List to save found elements.
+       var raycastResultList = new List<RaycastResult>();
+
+       // Find elements at pointerEventData position then save to raycastList.
+       graphicRaycaster.Raycast(pointerEventData, raycastResultList);
+
+       // Get buttons from saved list.
+       var raycastResult = raycastResultList.Find(element => element.gameObject.GetComponent<Button>());
+
+       // Return null if no button found.
+       if (raycastResult.gameObject == null)
+       {
+           return null;
+       }
+
+       // Otherwise return found button.
+       return raycastResult.gameObject.GetComponent<Button>();
    }
Enter fullscreen mode Exit fullscreen mode

Here, finding a button with a click position is done.

Call detect function

When the OpenVR mouse event is dispatched, retrieve the mouse position and pass it to the GetButtonByPosition() just created above.
There is no mouse click event in the OpenVR overlay event, so we will use MouseDown forclick detection.

void Update()
{
    Overlay.SetOverlayRenderTexture(dashboardHandle, renderTexture);

    var vrEvent = new VREvent_t();
    var uncbVREvent = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(VREvent_t));
    while (OpenVR.Overlay.PollNextOverlayEvent(dashboardHandle, ref vrEvent, uncbVREvent))
    {
        switch (vrEvent.eventType)
        {
-           // Remove MouseUp event we don't use this time.
-           case (uint)EVREventType.VREvent_MouseButtonDown:
-               Debug.Log($"MouseDown: ({vrEvent.data.mouse.x}, {vrEvent.data.mouse.y})");
-               break;

            case (uint)EVREventType.VREvent_MouseButtonUp:
-               Debug.Log($"MouseUp: ({vrEvent.data.mouse.x}, {vrEvent.data.mouse.y})");
+               var button = GetButtonByPosition(new Vector2(vrEvent.data.mouse.x, vrEvent.data.mouse.y));
+               Debug.Log(button);
                break;
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Run the program, and click the buttons on the dashboard. The button name should be logged into the Unity console. Currently, it displays the wrong button name but it’s OK for now.
If we click an empty area in the overlay, it should say Null.

Image description

Flip the click position vertically

Why the clicked button name is swapped to the other one is that the mouse position Y-axis (multiplied mouse scaling factor to V value) is reversed between OpenVR overlay and Unity canvas.

OpenVR overlay mouse position is notified as the left bottom is (0, 0). This is originally the same as Unity.
But the notified mouse position is flipped vertically this time because we have flipped the V-axis with SetOverlayTextureBounds() when drawing texture to avoid flipping vertically caused by the UV coordinate difference between DirectX and Unity. This texture bounds change affects the notified mouse position.

In this tutorial, we assume that the graphics API is always DirectX, and always flips texture with SetOverlayTextureBounds(), so we will flip the click position V (Y) axis at the event handling.

void Update()
{
    Overlay.SetOverlayRenderTexture(dashboardHandle, renderTexture);

    var vrEvent = new VREvent_t();
    var uncbVREvent = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(VREvent_t));
    while (OpenVR.Overlay.PollNextOverlayEvent(dashboardHandle, ref vrEvent, uncbVREvent))
    {
        switch (vrEvent.eventType)
        {
            case (uint)EVREventType.VREvent_MouseButtonUp:
+               vrEvent.data.mouse.y = renderTexture.height - vrEvent.data.mouse.y;
                var button = GetButtonByPosition(new Vector2(vrEvent.data.mouse.x, vrEvent.data.mouse.y));
                Debug.Log(button);
                break;
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Run the program, click the Left Hand and Right Hand buttons, and check the correct button is captured.

Image description

Switch controllers by clicking buttons

Now, we have got which button is clicked. Next, switch which hand to display the current time when a button is clicked.
We have created the code to switch hands and attached it to each Button's onClick method, so we will just call it.

DashboardOverlay.cs

void Update()
{
    Overlay.SetOverlayRenderTexture(dashboardHandle, renderTexture);

    var vrEvent = new VREvent_t();
    var uncbVREvent = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(VREvent_t));
    while (OpenVR.Overlay.PollNextOverlayEvent(dashboardHandle, ref vrEvent, uncbVREvent))
    {
        switch (vrEvent.eventType)
        {
            case (uint)EVREventType.VREvent_MouseButtonUp:
                vrEvent.data.mouse.y = renderTexture.height - vrEvent.data.mouse.y;
                var button = GetButtonByPosition(new Vector2(vrEvent.data.mouse.x, vrEvent.data.mouse.y));
-               Debug.Log(button);
+               if (button != null)
+               {
+                   button.onClick.Invoke();
+               }
                break;
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

Run the program, open the dashboard then click buttons.
It should switch which hand to show the current time.

Image description

Here, creating dashboard overlay and switch controllers are done.

Organize code

Mouse Scaling Factor

Move the mouse scaling factor setting as SetOverlayMouseScale().

OpenVRUtil.cs

...

public static void SetOverlayRenderTexture(ulong handle, RenderTexture renderTexture)
{
    if (!renderTexture.IsCreated())
    {
        return;
    }

    var nativeTexturePtr = renderTexture.GetNativeTexturePtr();
    var texture = new Texture_t
    {
        eColorSpace = EColorSpace.Auto,
        eType = ETextureType.DirectX,
        handle = nativeTexturePtr
    };
    var error = OpenVR.Overlay.SetOverlayTexture(handle, ref texture);
    if (error != EVROverlayError.None)
    {
        throw new Exception("Failed to draw texture: " + error);
    }
}

+ public static void SetOverlayMouseScale(ulong handle, int x, int y)
+ {
+     var pvecMouseScale = new HmdVector2_t()
+     {
+         v0 = x,
+         v1 = y
+     };
+     var error = OpenVR.Overlay.SetOverlayMouseScale(handle, ref pvecMouseScale);
+     if (error != EVROverlayError.None)
+     {
+         throw new Exception("Failed to set mouse scale: " + error);
+     }
+ }

...
Enter fullscreen mode Exit fullscreen mode

DashboardOverlay.cs

private void Start()
{
    OpenVRUtil.System.InitOpenVR();

    (dashboardHandle, thumbnailHandle) = Overlay.CreateDashboardOverlay("WatchDashboardKey", "Watch Setting");

    var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
    Overlay.SetOverlayFromFile(thumbnailHandle, filePath);

    Overlay.SetOverlaySize(dashboardHandle, 2.5f);
    Overlay.FlipOverlayVertical(dashboardHandle);

-   var pvecMouseScale = new HmdVector2_t()
-   {
-       v0 = renderTexture.width,
-       v1 = renderTexture.height
-   };
-   var error = OpenVR.Overlay.SetOverlayMouseScale(dashboardHandle, ref pvecMouseScale);
-   if (error != EVROverlayError.None)
-   {
-       throw new Exception("Failed to set mouse scale: " + error);
-   }
+   Overlay.SetOverlayMouseScale(dashboardHandle, renderTexture.width, renderTexture.height);
}
Enter fullscreen mode Exit fullscreen mode

Event handling

Move the event handling as ProcessOverlayEvents().

DashboardOverlay.cs

void Update()
{
    Overlay.SetOverlayRenderTexture(dashboardHandle, renderTexture);
-
-   var vrEvent = new VREvent_t();
-   var uncbVREvent = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(VREvent_t));
-   while (OpenVR.Overlay.PollNextOverlayEvent(dashboardHandle, ref vrEvent, uncbVREvent))
-   {
-       switch (vrEvent.eventType)
-       {
-           case (uint)EVREventType.VREvent_MouseButtonUp:
-               vrEvent.data.mouse.y = renderTexture.height - vrEvent.data.mouse.y;
-               var button = GetButtonByPosition(vrEvent.data.mouse.x, vrEvent.data.mouse.y);
-               if (button != null)
-               {
-                   button.onClick.Invoke();
-               }
-               break;
-       }
-  }
+  ProcessOverlayEvents();
}

...

+ private void ProcessOverlayEvents()
+ {
+     var vrEvent = new VREvent_t();
+     var uncbVREvent = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(VREvent_t));
+     while (OpenVR.Overlay.PollNextOverlayEvent(dashboardHandle, ref vrEvent, uncbVREvent))
+     {
+         switch (vrEvent.eventType)
+         {
+             case (uint)EVREventType.VREvent_MouseButtonUp:
+                 vrEvent.data.mouse.y = renderTexture.height - vrEvent.data.mouse.y;
+                 var button = GetButtonByPosition(new Vector2(vrEvent.data.mouse.x, vrEvent.data.mouse.y));
+                 if (button != null)
+                 {
+                     button.onClick.Invoke();
+                 }
+                 break;
+         }
+     }
+ }

private Button GetButtonByPosition(float x, float y)
{
    ...
Enter fullscreen mode Exit fullscreen mode

Final code

WatchOverlay.cs

using UnityEngine;
using Valve.VR;
using System;
using OpenVRUtil;

public class WatchOverlay : MonoBehaviour
{
    public Camera camera;
    public RenderTexture renderTexture;
    public ETrackedControllerRole targetHand = ETrackedControllerRole.LeftHand;
    private ulong overlayHandle = OpenVR.k_ulOverlayHandleInvalid;

    [Range(0, 0.5f)] public float size;

    [Range(-0.2f, 0.2f)] public float leftX;
    [Range(-0.2f, 0.2f)] public float leftY;
    [Range(-0.2f, 0.2f)] public float leftZ;
    [Range(0, 360)] public int leftRotationX;
    [Range(0, 360)] public int leftRotationY;
    [Range(0, 360)] public int leftRotationZ;

    [Range(-0.2f, 0.2f)] public float rightX;
    [Range(-0.2f, 0.2f)] public float rightY;
    [Range(-0.2f, 0.2f)] public float rightZ;
    [Range(0, 360)] public int rightRotationX;
    [Range(0, 360)] public int rightRotationY;
    [Range(0, 360)] public int rightRotationZ;

    private void Start()
    {
        OpenVRUtil.System.InitOpenVR();
        overlayHandle = Overlay.CreateOverlay("WatchOverlayKey", "WatchOverlay");

        Overlay.FlipOverlayVertical(overlayHandle);
        Overlay.SetOverlaySize(overlayHandle, size);
        Overlay.ShowOverlay(overlayHandle);
    }

    private void Update()
    {
        Vector3 position;
        Quaternion rotation;

        if (targetHand == ETrackedControllerRole.LeftHand)
        {
            position = new Vector3(leftX, leftY, leftZ);
            rotation = Quaternion.Euler(leftRotationX, leftRotationY, leftRotationZ);
        }
        else
        {
            position = new Vector3(rightX, rightY, rightZ);
            rotation = Quaternion.Euler(rightRotationX, rightRotationY, rightRotationZ);
        }

        var controllerIndex = OpenVR.System.GetTrackedDeviceIndexForControllerRole(targetHand);
        if (controllerIndex != OpenVR.k_unTrackedDeviceIndexInvalid)
        {
            Overlay.SetOverlayTransformRelative(overlayHandle, controllerIndex, position, rotation);
        }

        Overlay.SetOverlayRenderTexture(overlayHandle, renderTexture);
    }

    private void OnApplicationQuit()
    {
        Overlay.DestroyOverlay(overlayHandle);
    }

    private void OnDestroy()
    {
        OpenVRUtil.System.ShutdownOpenVR();
    }
}
Enter fullscreen mode Exit fullscreen mode

DashboardOverlay.cs

using UnityEngine;
using Valve.VR;
using System;
using OpenVRUtil;
using UnityEngine.UI;
using UnityEngine.EventSystems;
using System.Collections.Generic;

public class DashboardOverlay : MonoBehaviour
{
    public Camera camera;
    public RenderTexture renderTexture;
    public GraphicRaycaster graphicRaycaster;
    public EventSystem eventSystem;
    private ulong dashboardHandle = OpenVR.k_ulOverlayHandleInvalid;
    private ulong thumbnailHandle = OpenVR.k_ulOverlayHandleInvalid;

    private void Start()
    {
        OpenVRUtil.System.InitOpenVR();

        (dashboardHandle, thumbnailHandle) = Overlay.CreateDashboardOverlay("WatchDashboardKey", "Watch Setting");

        var filePath = Application.streamingAssetsPath + "/sns-icon.jpg";
        Overlay.SetOverlayFromFile(thumbnailHandle, filePath);

        Overlay.FlipOverlayVertical(dashboardHandle);
        Overlay.SetOverlaySize(dashboardHandle, 2.5f);
        Overlay.SetOverlayMouseScale(dashboardHandle, renderTexture.width, renderTexture.height);
    }

    private void Update()
    {
        Overlay.SetOverlayRenderTexture(dashboardHandle, renderTexture);
        ProcessOverlayEvents();
    }

    private void OnApplicationQuit()
    {
        Overlay.DestroyOverlay(dashboardHandle);
    }

    private void OnDestroy()
    {
        OpenVRUtil.System.ShutdownOpenVR();
    }

    private void ProcessOverlayEvents()
    {
        var vrEvent = new VREvent_t();
        var uncbVREvent = (uint)System.Runtime.InteropServices.Marshal.SizeOf(typeof(VREvent_t));
        while (OpenVR.Overlay.PollNextOverlayEvent(dashboardHandle, ref vrEvent, uncbVREvent))
        {
            switch (vrEvent.eventType)
            {
                case (uint)EVREventType.VREvent_MouseButtonUp:
                    vrEvent.data.mouse.y = renderTexture.height - vrEvent.data.mouse.y;
                    var button = GetButtonByPosition(new Vector2(vrEvent.data.mouse.x, vrEvent.data.mouse.y));
                    if (button != null)
                    {
                        button.onClick.Invoke();
                    }
                    break;
            }
        }
    }

    private Button GetButtonByPosition(Vector2 position)
    {
        var pointerEventData = new PointerEventData(eventSystem);
        pointerEventData.position = new Vector2(position.x, position.y);

        var raycastResultList = new List<RaycastResult>();
        graphicRaycaster.Raycast(pointerEventData, raycastResultList);

        var raycastResult = raycastResultList.Find(element => element.gameObject.GetComponent<Button>());
        if (raycastResult.gameObject == null)
        {
            return null;
        }
        return raycastResult.gameObject.GetComponent<Button>();
    }
}
Enter fullscreen mode Exit fullscreen mode

OpenVRUtil.cs

using UnityEngine;
using Valve.VR;
using System;

namespace OpenVRUtil
{
    public static class System
    {
        public static void InitOpenVR()
        {
            if (OpenVR.System != null) return;

            var error = EVRInitError.None;
            OpenVR.Init(ref error, EVRApplicationType.VRApplication_Overlay);
            if (error != EVRInitError.None)
            {
                throw new Exception("Failed to initialize OpenVR: " + error);
            }
        }

        public static void ShutdownOpenVR()
        {
            if (OpenVR.System != null)
            {
                OpenVR.Shutdown();
            }
        }
    }

    public static class Overlay
    {
        public static ulong CreateOverlay(string key, string name)
        {
            var handle = OpenVR.k_ulOverlayHandleInvalid;
            var error = OpenVR.Overlay.CreateOverlay(key, name, ref handle);
            if (error != EVROverlayError.None)
            {
                throw new Exception("Failed to create overlay: " + error);
            }

            return handle;
        }

        public static (ulong, ulong) CreateDashboardOverlay(string key, string name)
        {
            ulong dashboardHandle = 0;
            ulong thumbnailHandle = 0;
            var error = OpenVR.Overlay.CreateDashboardOverlay(key, name, ref dashboardHandle, ref thumbnailHandle);
            if (error != EVROverlayError.None)
            {
                throw new Exception("Failed to create dashboard overlay: " + error);
            }

            return (dashboardHandle, thumbnailHandle);
        }

        public static void DestroyOverlay(ulong handle)
        {
            if (handle != OpenVR.k_ulOverlayHandleInvalid)
            {
                var error = OpenVR.Overlay.DestroyOverlay(handle);
                if (error != EVROverlayError.None)
                {
                    throw new Exception("Failed to dispose overlay: " + error);
                }
            }
        }

        public static void SetOverlayFromFile(ulong handle, string path)
        {
            var error = OpenVR.Overlay.SetOverlayFromFile(handle, path);
            if (error != EVROverlayError.None)
            {
                throw new Exception("Failed to draw image file: " + error);
            }
        }

        public static void ShowOverlay(ulong handle)
        {
            var error = OpenVR.Overlay.ShowOverlay(handle);
            if (error != EVROverlayError.None)
            {
                throw new Exception("Failed to show overlay: " + error);
            }
        }

        public static void SetOverlaySize(ulong handle, float size)
        {
            var error = OpenVR.Overlay.SetOverlayWidthInMeters(handle, size);
            if (error != EVROverlayError.None)
            {
                throw new Exception("Failed to set overlay size: " + error);
            }
        }

        public static void SetOverlayTransformAbsolute(ulong handle, Vector3 position, Quaternion rotation)
        {
            var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
            var matrix = rigidTransform.ToHmdMatrix34();
            var error = OpenVR.Overlay.SetOverlayTransformAbsolute(handle, ETrackingUniverseOrigin.TrackingUniverseStanding, ref matrix);
            if (error != EVROverlayError.None)
            {
                throw new Exception("Failed to set overlay position: " + error);
            }
        }

        public static void SetOverlayTransformRelative(ulong handle, uint deviceIndex, Vector3 position, Quaternion rotation)
        {
            var rigidTransform = new SteamVR_Utils.RigidTransform(position, rotation);
            var matrix = rigidTransform.ToHmdMatrix34();
            var error = OpenVR.Overlay.SetOverlayTransformTrackedDeviceRelative(handle, deviceIndex, ref matrix);
            if (error != EVROverlayError.None)
            {
                throw new Exception("Failed to set overlay position: " + error);
            }
        }

        public static void FlipOverlayVertical(ulong handle)
        {
            var bounds = new VRTextureBounds_t
            {
                uMin = 0,
                uMax = 1,
                vMin = 1,
                vMax = 0
            };

            var error = OpenVR.Overlay.SetOverlayTextureBounds(handle, ref bounds);
            if (error != EVROverlayError.None)
            {
                throw new Exception("Failed to texture flip: " + error);
            }
        }

        public static void SetOverlayRenderTexture(ulong handle, RenderTexture renderTexture)
        {
            if (!renderTexture.IsCreated()) return;

            var nativeTexturePtr = renderTexture.GetNativeTexturePtr();
            var texture = new Texture_t
            {
                eColorSpace = EColorSpace.Auto,
                eType = ETextureType.DirectX,
                handle = nativeTexturePtr
            };
            var error = OpenVR.Overlay.SetOverlayTexture(handle, ref texture);
            if (error != EVROverlayError.None)
            {
                throw new Exception("Failed to draw texture: " + error);
            }
        }

        public static void SetOverlayMouseScale(ulong handle, int x, int y)
        {
            var pvecMouseScale = new HmdVector2_t()
            {
                v0 = x,
                v1 = y
            };
            var error = OpenVR.Overlay.SetOverlayMouseScale(handle, ref pvecMouseScale);
            if (error != EVROverlayError.None)
            {
                throw new Exception("Failed to set mouse scale: " + error);
            }
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

WatchSettingController.cs

using UnityEngine;
using Valve.VR;

public class WatchSettingController : MonoBehaviour
{
    [SerializeField] private WatchOverlay watchOverlay;

    public void OnLeftHandButtonClick()
    {
        watchOverlay.targetHand = ETrackedControllerRole.LeftHand;
    }

    public void OnRightHandButtonClick()
    {
        watchOverlay.targetHand = ETrackedControllerRole.RightHand;
    }
}
Enter fullscreen mode Exit fullscreen mode

Here, we made it to deal OpenVR overlay event.
The final task is to display the current time in a few seconds when the controller button is pushed.

Top comments (0)

Read next

vinnugollakoti profile image

Day 41 journal

eashwar ammu -

reegan profile image

Day 10 Task:

AmalaReegan -

igorbenav profile image

Building a Local AI Task Planner with ClientAI and Ollama

Igor Benav -

vinnugollakoti profile image

Day 36 Journal

eashwar ammu -