Welcome to the Exciting World of WebXR and Babylon.js

This article series serves as a first step into WebXR development using Babylon.js, a powerful and user-friendly framework. This journey will help you make your first steps into XR on the web.

ℹ️ The goal of this series is for you to find yourself immersed in an interactive mixed reality scene. Each article is building up on each other. You’ll tackle every aspect needed, to create the final scene.

Let’s dive in!

What Is WebXR?

WebXR is a technology that allows web developers to create immersive 3D XR experiences directly in web browsers. It's a game-changer for the field of virtual and augmented reality, making these experiences more accessible than ever before.

Browser support

According to caniuse WebXR is still widely unsupported and if supported, only partially. Safest bet currently is using either Chrome or a Chromium based browser (Edge, Arc, Opera)

Why Babylon.js?

Babylon.js stands out as a robust framework for building 3D XR experiences. Its ease of use and powerful features make it an ideal choice for developers venturing into the world of WebXR.

Your Learning Journey

📚 The best way to go through each article is by starting the related code part and either viewing it in the browser or directly on the Meta Quest 3.

In this series, we'll start with the fundamentals of WebXR programming and gradually move to more advanced techniques. Each article will provide practical guidance and insights into AR development. You'll learn how to:

• Set up your development environment.
• Understand and use Babylon.js core features like:
  • Plane detection
  • Meshes & Materials
  • Hit-Testing
  • Input & Controllers & Ray Casting
  • Animation
  • Anchors
  • Models & Assets
• Create an immersive and interactive 3D scene.

Prerequisites

ℹ️ Before diving in, make sure you have

• Node und npm installed (v20.9.0 - 11/2023)
• A basic understanding of Javascript and Typescript
• A MetaQuest 3 Headset

📚 Because of the nature of Babylon.js being a game engine, we will come across terms like vertices, polygons and meshes as well as some geometry. So a little background on these topics will be helpful but not needed.

Setting Up Your Environment

Here's a step-by-step guide to get your development environment ready:

ℹ️ The base project can be found here: Github/webxr-article-series

1. Download the Base Project: Start by downloading our base project specifically designed for WebXR and Babylon.js. This will give you a base project to work with.

2. Install Dependencies: Open your terminal, navigate to your project's directory, and run npm install. This command will download and install all the necessary dependencies for your project.

3. Start Your Webpack Server: Run npm start to fire up your Webpack server. Webpack is a module bundler that will help in efficiently compiling and running your code.
   This will load the first part of code for this article.
   To load any other part run npm start --part=[1-10].

4. Accessing Your Project on XR Device: Use LocalTunnel to expose your local web server to the internet. This will allow your XR device to access your project. Simply run npx localtunnel --port 8080, replacing 8080 with your server's port number (Webpack uses 8080 per default).

ℹ️ LocalTunnel can be very slow at times. An alternative is TunnelMole.

Testing Your Setup

❗️ The first time you access the site provided by LocalTunnel, you will land on a page similar to the following. For security reasons, we have to enter our IP address in order to be able to access our local web server. You can view your IP at https://ipv4.icanhazip.com/. Copy this into the input field for Endpoint IP and confirm by clicking on the button.

Once everything is set up, you should be able to access your project through a public URL provided by LocalTunnel. Open the link provided by LocalTunnel (shown in your terminal) in a browser of your choice.

❗️ The page load can be slow through LocalTunnel. The scene will always load eventually.

Test this link on your MetaQuest 3 headset or any other AR-enabled device to ensure everything is working correctly. You should see a page similar to the one below. To enter the XR scene, simply click on the button on the bottom right.

📚 To exit an XR session in the Meta Quest 3 headset, you have to press the A or X button on either of the controllers and choose Quit to end the session. You’ll be going back to your browser window.

This is also needed every time you make any code changes or when re-/starting Webpack.

The immersive web emulator for the browser

📚 If you don't have a device at hand, you can install Meta's Immersive Web Emulator as a browser extension for Chrome and Chromium-based browsers.

Immersive Web Emulator

The base template

ℹ️ The base template will always be your point of reference for the whole journey. Each part of the series will build up on the prior and will finish with a link to the then current state of our project.

You can find the different parts called index_[part].ts under the src folder

Let’s begin by going through the base template step by step. We will start seeing actual progress in the scenes composition in the next article. This part is to make you familiar with the base construct creating an XR scene in Babylon.js.

Importing the Babylonjs dependencies and the debugger.
 

import {
    Engine,
    Scene,
    Vector3,
    WebXRDefaultExperience,
    ShadowGenerator,
    DirectionalLight,
    IShadowLight,
    HemisphericLight,
} from '@babylonjs/core';

import { Inspector } from '@babylonjs/inspector';

Defining possible session modes for WebXR.

type SessionModes = "immersive-ar" | "immersive-vr" | "inline";

These modes dictate how the content will be presented to the user and what kind of input and output capabilities are available. The primary session modes in WebXR are:

1. Immersive VR: This mode is designed for fully immersive VR experiences. When a session is started in immersive VR mode, it takes over the entire display, blocking out the real world entirely. This mode is typically used with VR headsets, providing a full 360-degree view of a virtual environment. It's ideal for games, simulations, and other applications where complete immersion in a virtual world is desired.

2. Immersive AR: This mode is for augmented reality experiences. Unlike immersive VR, immersive AR allows the user to see the real world around them, with digital content superimposed. This mode is used with AR-capable devices, like some smartphones and dedicated AR headsets. It's suitable for applications that augment the real world with additional information or virtual objects, such as navigation aids, information overlays, or interactive games that integrate with the physical environment.

3. Inline: This mode is for non-immersive experiences that are displayed within a standard web page, typically in a canvas element. The inline session mode doesn’t take over the full display and is not exclusive to AR or VR. It's useful for 3D rendering that integrates with other web content, like product previews or simple interactive 3D models. You don't need any special glasses or headsets to view it. It's like seeing a 3D model or interactive scene appear on a regular webpage.

Defining possible reference space types for WebXR.

These reference space types are crucial for creating immersive and interactive 3D experiences in a virtual environment. The main reference space types in WebXR are:

Creating our main class and variables

class XrExperience {
    // The canvas element is used to display the scene.
    _canvas: HTMLCanvasElement;

    // The 3D engine is responsible for creating and updating the scene.
    _engine: Engine;

    // The scene is the 3D environment where the 3D models are displayed.
    _scene: Scene;

    // debug mode
    _debug: boolean;

    // xrExperience helper
    _xr: WebXRDefaultExperience | null;

    // xrExperience options
    _sessionMode: SessionModes;
    _referenceSpaceType: ReferenceSpaceType;
    _optionalFeatures: boolean;

The constructor is setting up the necessary elements for a 3D scene and configuring the WebXR experience.

Error Handling for WebGL Support

if (!Engine.isSupported()) {
    throw 'WebGL not supported';
}

This segment checks if WebGL is supported in the user's browser. WebGL (Web Graphics Library) is essential for rendering 3D graphics in web browsers. If WebGL is not supported (Engine.isSupported() returns false), the constructor throws an error, stopping further execution. This is important for ensuring that the application only runs in environments where it can function correctly.

Setting Up the Canvas and Engine

this._canvas = document.getElementById('canvas') as HTMLCanvasElement;
this._engine = new Engine(this._canvas, true);

Here, the constructor locates a canvas element in the DOM (Document Object Model) with the ID renderCanvas and assigns it to this._canvas. The canvas is where the 3D scene will be rendered. Then, it creates a new instance of the Babylon.js Engine, passing the canvas as an argument. This engine is responsible for the rendering and overall management of the 3D scene.

Creating the Scene

this._scene = new Scene(this._engine);

This line initializes a new Scene object using the engine. The scene is where all the 3D objects, lights, cameras, and other elements will be added.

Debugging and XR Configuration

this._debug = true;
this._sessionMode = "immersive-ar";
this._referenceSpaceType = "local-floor";
this._optionalFeatures = true;

The constructor sets up some properties for debugging and configuring the WebXR experience. It sets the session mode to immersive-ar (augmented reality) and the reference space type to local-floor, indicating the type of XR experience and the reference space for tracking.

Initializing the XR Experience

this._xr = null;
this.createXrExperience().then(() => {
    this.createScene().then(() => {
        ...
    });
}).catch((error) => {
    console.log(error);
});

These lines initialize the XR experience and the scene creation. The createXrExperience and createScene methods are custom methods in the class that set up the XR experience and populate the scene with content.

Render Loop and Resize Event Handling

this._engine.runRenderLoop(() => {
    this._scene.render();
});
window.addEventListener('resize', () => {
    this._engine.resize();
});

The runRenderLoop method of the engine is called with a function that renders the scene. This creates an infinite loop that continuously renders the scene, allowing for animations and real-time updates. Additionally, an event listener is added to the window object to handle resize events, ensuring that the engine adjusts its rendering to fit the new window size.

Creating a scene

This creates the scene and sets up lights and shadows. This is also the point where the debug inspector can be enabled.

async createScene(): Promise<Scene> {

  this.createLightsAndShadows();

  if (this._debug) Inspector.Show(this._scene, {}); // enable for debugging

  return this._scene;
}

Creating Lights and Shadows

First 2 different types of lights are created to support the scene.

createLightsAndShadows() {
  const lights = this.createLights();

  const shadowGenerator = new ShadowGenerator(1024, lights as IShadowLight);

  shadowGenerator.useBlurExponentialShadowMap = true;
  shadowGenerator.blurKernel = 32;
}

createLights() {

  const directionalLight = new DirectionalLight("directionalLight", new Vector3(0, 1, 1), this._scene);
  directionalLight.intensity = 0.7;
  directionalLight.position = new Vector3(0, 2, 0);

  const hemiLight = new HemisphericLight("hemisphericLight", new Vector3(0, 1, 0), this._scene);
  hemiLight.intensity = 0.7;

  return directionalLight;
}

• Directional Light: A DirectionalLight with a specified direction (new Vector3(0, 1, 1)), indicating the direction the light is coming from. Its intensity is set to 0.7, and it's positioned at new Vector3(0, 2, 0) in the scene.

• Hemispheric Light: A HemisphericLight is also created. This type of light emits light from the direction of the specified vector (new Vector3(0, 1, 0)) and lights the scene with a gradient from the sky (light) to the ground (dark). Its intensity is similarly set to 0.7.

Then a ShadowGenerator is instantiated to create a shadow with a map size of 1024. The size of the shadow map affects the quality and performance of the shadow rendering.

The shadow generator's properties are set to use a blur exponential shadow map (useBlurExponentialShadowMap = true) and a blur kernel size of 32. These settings enhance the visual quality of the shadows, making them softer and more realistic by reducing artefacts like aliasing (with the help of exponential shadow mapping) and blurring to further smoothing the shadows.

Starting point of our application.

new XrExperience();

Conclusion

This concludes the first part of our journey into WebXR and Babylon.js, offering a solid foundation in developing AR experiences. This sets the stage for continued learning and development in this dynamic field, inviting you to stay tuned for further insights and enhancements in web-based mixed reality.

The second part of our journey will be about Planes in WebXR.