Part 4: Hit Testing (WebXR with Babylon.js)

👀 Stumbled here on accident? Start with the first part!

Welcome back to the next part of our series. This time we learn about Hit Testing.

ℹ️ Remember - you can always run the code associated with this article and follow along using

npm start --part=4

What is a Hit Test?

ℹ️ A hit test in WebXR is typically used to determine where in the real world a virtual object should be placed or how it should interact with the real environment.

📚 We will achieve this by creating a torus mesh (looks like a Donut 🍩) that’ll show us on which planes a hit can be detected and where virtual objects can be placed in accordance to the physical environment. Like placing an object on the floor, or on a detected table.

How it works

A hit test involves sending a virtual ray from a specific point (often the user's viewpoint or a point in virtual space) and checking where this ray intersects with real-world surfaces.

📚 While it is common to use the camera (representing the user's view in most cases) as the origin for hit tests, especially in scenarios like first-person views or camera-based selections, Babylon.js is flexible enough to allow hit tests to be initiated from any point in the 3D space.

Use cases

Camera-Based Hit Testing: The most straightforward and common scenario, where the hit test is performed from the camera's viewpoint. This is typically used for selecting or interacting with objects in the scene based on the user's current view.

Arbitrary Point in Space: Developers can initiate a hit test from any arbitrary point in the 3D space, not just the camera. This is useful for scenarios where you need to check for intersections or collisions at specific locations in the scene, independent of the camera's position.

Moving Objects: Hit tests can also be initiated from moving objects within the scene, like a character's hand or a moving vehicle, to check for interactions or collisions with other objects in the scene.

📚 The choice of the origin for a hit test depends on the specific requirements of the application being developed.

ℹ️ Hit testing often relies on the detection of planes or surfaces in the real world to place virtual objects accurately.

Adding a Mesh as a marker

First we create a Mesh in form of a torus that serves us as a visual marker for a hit test.

addMarkerForHitTest() {
    this._marker = MeshBuilder.CreateTorus("marker", { diameter: 0.15, thickness: 0.05 }, this._scene);
    this._marker.isVisible = false;
    this._marker.receiveShadows = true;
    this._marker.checkCollisions = true;
    this._marker.rotationQuaternion = new Quaternion();
}

Performing a Hit Test

Next we perform hit testing to determine the interaction between the real world and virtual objects.

addFeaturesToSession() {
  ...
  try {
    ...
    this._xrHitTest = this._fm.enableFeature(WebXRFeatureName.HIT_TEST, "latest") as WebXRHitTest;
  } catch (error) {
    ...
  }
  ...
}

First we initialize and enable the hit testing feature in Babylon.js by adding it as a feature to our session.

performHitTest() {
    if (this._xrHitTest === null || this._marker === null) {
        return;
    }
    this._xrHitTest.onHitTestResultObservable.add((results) => {
        if (results.length) {
            this._marker!.isVisible = true;
            this._hitTest = results[0];
            this._hitTest.transformationMatrix.decompose(undefined, this._marker!.rotationQuaternion!, this._marker!.position);
        } else {
            this._marker!.isVisible = false;
            this._hitTest = undefined;
        }
    });
}

As soon as hit test results are available, the onHitTestResultObservable.add observer method is called.

When hit test results are present (results.length > 0):

• Makes the marker visible.

• Stores the first hit test result (results[0]) in a local variable (_hitTest).

• Uses the transformation matrix of the hit test result to update the position and orientation of the marker in the scene. This positions and aligns the marker at the point where the hit test detected a collision with a real surface.

If no hit test results are present, the marker is made invisible again and the local variable _hitTest is reset.

Adding Hit Testing to the scene

async createScene(): Promise<Scene> {
  ...
  this.addMarkerForHitTest();
  this.performHitTest();

  return this._scene;
}

Once again, we have to add the addMarkerForHitTestand the performHitTest functions to the scene.

Conclusion

In summary, hit testing in WebXR, particularly with Babylon.js, is an essential tool for creating interactive augmented reality experiences. It allows developers to place virtual objects accurately within the real world, enhancing the realism and immersion of XR applications. The ability to initiate hit tests from different points, like the user’s camera or specific locations in 3D space, provides versatility in application design. Understanding and applying hit testing is key to develop engaging and believable content.

In the fifth part of this series we take a look at input controllers and ray casting.