Creating a HUD in react-three-fiber

Follow along as I try building a virtual reality application using react-three-fiber. See my fist post on building the scene.

After creating a basic scene with react-three-fiber, I wanted to add a HUD for the user.

Single Camera

On my first several attempts, I tried to render multiple cameras, a PerspectiveCamera for the player and an OrthographicCamera for the HUD. After thinking about targeting a virtual-reality context, I realized I needed a spatial UI.

use the mouse to drag the camera and look around

Having the HUD follow the camera.

Instead of sticking the HUD directly to the camera, I used react-spring/three to animate the HUD's group with a spring to bounce to the camera orientation.

Inside of the useFrame hook, I grabbed the camera's position and orientation. Using the set provided by useSpring, I update the HUD's target position and rotation.

Next Steps

HUD positioning

I need to figure out how to get the headset position and place the HUD based on that. Simply using the PerspectiveCamera position doesn't seem to place it HUD based on the headset position, but maybe the camera's position is the play area?

HUD interaction

Now that I have a HUD, I will need to to allow the player to interact with the HUD and figure out a layout.