Virtual and Augmented Reality - Is It the Future Already?

VR headsets and virtual HUDs were once mere science fiction, but both exist in modern day! Are these burgeoning fields ripe for innovation, or just a passing fad? And how does one code VR/AR apps anyway?

Virtual Reality

VR has come a long way from the bulky VR arcade machines that some lucky few may have crawled into in the early 90's. Today, there exist a few different VR headset options for home use such as the HTC Vive or Oculus Quest, each costing about as much or even less than a modern gaming console. While these headsets do offer many different video game options such as the well known rhythm game Beat Saber (pictured above), there are other virtual experiences users can enjoy. For instance, many use the popular chat room like VR Chat program for online socialization, where they are able to customize virtual environments or even personal avatars for others to perceive them as. Another example is Google Earth VR, which allows one the experience of standing on any street corner in the world, or even soaring through the air like a superhero!

VR is not without its flops, of course. Some of these, like the Virtual Boy, were simply too ambitious for the current level of tech. One of the most infamous modern VR flops, however, would be Facebook's "Metaverse", which was designed to be an immersive virtual world which users could shop, play, and even work in. Unfortunately, it was marred by poor design decisions, numerous delays, and a stunning lack of critical features. According to Slash Gear, Metaverse currently sees only about 400 users logged on at any given time, far from the revolutionary product it was pitched as.

Augmented Reality

While Virtual Reality seeks to create immersive virtual environments for users to interact with, Augmented Reality instead seeks to provide an altered view of the real world, usually through some sort of lens or overlay. Nowadays, this is most commonly experienced through a Smartphone, altering the view through the device's camera in some way. One of the most popular AR Smartphone apps is the well known Pokemon Go game, which allows users to find and capture various creatures by physically moving around in the real world. The widely used app Snapchat also sports popular AR technology in the form of various camera lenses which alter faces and/or environments in different ways.

AR has some practical applications as well, though many of these are still new and not necessarily very widely available. One example of this would be in the automotive industry, where some new models of cars such as the 2022 Mercedes-Benz EQS have incorporated virtual overlays over video feeds on the dashboard or even the windshields of the vehicles themselves! These can communicate things like current speed, visual queues for GPS routes, and even highlighting the sides of the road at night or during visually impairing weather conditions like fog or heavy precipitation.

Of course, Augmented Reality is not without its flops as well. Google Glass is perhaps the most well known - a product designed to project a virtual overlay onto a pair of glasses worn by the user. Unfortunately, the enthusiasm Google devs had for the device was not shared by their customer base; the product was seen as silly looking and not particularly game changing - few people ever purchased one.

Under the Hood: How Does the Code Work?

While ultimately providing a different form of functionality (visually modifying the surrounding environment versus creating an entirely new one), both AR and VR operate using the same basic principal: tracking patterns in 3D space using specialized cameras. There are a few ways this is typically done. The most well known method is to use Image Targets, which are graphics which physically exist in the real world and which programs are able to recognize. The most ubiquitous example of these is QR codes, barcode-like graphics that are scannable with most smartphones and which typically provide small bits of data, such as the URL for a restaurant's menu, or an image to display on your phone. Another example would be the controllers for VR headsets, which usually use LED lights or other subtle features to allow the headset cameras to track them. More complicated than that is Plane Detection, which uses "feature points", patterns that weren't specifically placed to interact with the app. Examples of this would include face detection software as well as the feature of some VR headsets such as the Oculus Quest which detects hands, eliminating the need for controllers. Finally, there is Depth Sensoring, a cutting edge approach for AR which uses a device's depth sensor to measure distance, enabling the placement of virtual objects to be placed and tracked behind real world objects.

So what does the code actually look like for this sort of thing? Given that, as shown in the graph above, the most commonly used language for VR apps is JavaScript, we will look at some JavaScript code written by Minhaz which is used in conjunction with a little bit of HTML to recognize QR codes:

HTML:

<script 
    src="https://raw.githubusercontent.com/mebjas/html5-qrcode/master/minified/html5-qrcode.min.js">
</script>

<div id="reader"></div>

<input type="file" id="qr-input-file" accept="image/*">
<!-- 
  Or add captured if you only want to enable smartphone camera, PC browsers will ignore it.
-->

<input type="file" id="qr-input-file" accept="image/*" capture>

JavaScript:

/**
 * Scans an Image File for QR Code.
 * 
 * This feature is mutually exclusive to camera based scanning, you should call
 * stop() if the camera based scanning was ongoing.
 * 
 * @param {File} imageFile a local file with Image content.
 * @param {boolean} showImage if true the Image will be rendered on given element.
 * 
 * @returns Promise with decoded QR code string on success and error message on failure.
 *            Failure could happen due to different reasons:
 *            1. QR Code decode failed because enough patterns not found in image.
 *            2. Input file was not image or unable to load the image or other image load
 *              errors.
*/
scanFile(imageFile, showImage /* default = true */) {}

const html5QrCode = new Html5Qrcode(/* element id */ "reader");

// File based scanning
const fileinput = document.getElementById('qr-input-file');
fileinput.addEventListener('change', e => {
  if (e.target.files.length == 0) {
    // No file selected, ignore 
    return;
  }

  // Use the first item in the list
  const imageFile = e.target.files[0];
  html5QrCode.scanFile(imageFile, /* showImage= */true)
  .then(qrCodeMessage => {
    // success, use qrCodeMessage
    console.log(qrCodeMessage);
  })
  .catch(err => {
    // failure, handle it.
    console.log(`Error scanning file. Reason: ${err}`)
  });
});

html5QrCode.clear();

As explained by Minhaz in his blog post, the code above uses a div tag with the placeholder element "reader" in the HTML to hold the scanned QR code, which it reads from the input tag (note that only one of the two input tags in the HTML code block is meant to be included, depending on whether the app is meant to work only with smartphone cameras or if it is intended to work on PCs as well). The JavaScript code then uses an event listener to constantly scan image files for a recognizable QR code. Upon detecting one, it uses a Promise based API system to decode and display the encoded message (or throws an error if a problem occurs). Finally, html5QrCode.clear() is used to clear the canvas after use, as start() and stop() are not supported by the API used for this code.

The Future is Near!

While the world of VR and AR has had its share of gaffes, both in the past and the present, it definitely has more than a foothold in the modern tech industry. We may not be plugging in to the Matrix in the near future, but these technologies are slowly picking up steam and will likely continue their refinement. With the right innovation, they may even see an explosion in popularity that changes the game in the same way that smartphones, television, or even the internet once did!

Sources:
https://www.statista.com/statistics/1343292/coding-languages-used-in-ar-vr-worldwide/#:~:text=A%20survey%20conducted%20between%20late,projects%2C%20followed%20closely%20by%20Python.

https://gamedevacademy.org/how-to-code-an-ar-application/

https://www.youtube.com/watch?v=YaWzVkI4v0Q

https://spectrum.ieee.org/augmented-reality-car-hud?utm_campaign=post-teaser&utm_content=8k7cu2ho

https://virtualspeech.com/blog/history-of-vr

https://www.slashgear.com/1012807/the-biggest-vr-and-ar-flops-of-all-time/

https://techcrunch.com/2022/03/13/augmented-realitys-half-decade-of-stagnation/

https://www.freecodecamp.org/news/augmented-reality-with-javascript-a-case-study-c9cffaadcf07/

https://blog.minhazav.dev/HTML5-QR-Code-scanning-support-for-local-file-and-default-camera/

https://circuitstream.com/blog/vr-controllers-the-way-of-interacting-with-the-virtual-worlds