Virtual reality—making the unreal ‘real’
Expert reviewers
Essentials
- The concept of virtual reality (VR) has been around since the 1950s.
- VR aims to immerse a user in a computer-generated 3D environment so they feel part of that world.
- VR generally includes visual and audio simulation and, as the technology improves, touch, smell and the ability to affect and alter the virtual environment.
- There is still debate about what actually constitutes a true ‘VR’ experience.
- There is a wide spectrum of VR technologies, from cardboard cut-outs to fully equipped digital labs.
They were the buzzwords of 2016—virtual reality. Finally, the technology that had its origins in the 1950s was said to be about to live up to its long-held potential. As proof, a range of VR devices began to appear in shops and online aimed at the everyday consumer (or at least, the financially well-off consumer). How do these emerging consumer-directed VR technologies work, and are they really capable of making the unreal seem real?
What is virtual reality?
The term ‘virtual reality’ (VR) can be little confusing because it is currently used to describe many different experiences, from the basic—using just your computer or smartphone in a relatively passive manner—to complex whole body immersion in specialised rooms fitted out with tracking cameras, advanced haptics GLOSSARY hapticsthe science of applying touch (tactile) sensation and control to interaction with computer applications and real-time image rendering.
Most industry experts would say that a true VR experience is where the user feels a real sense of immersion in the computer-generated 3D environment, and is able to affect and interact with that environment. Depending on the quality of the system, users can experience the five senses of sight, hearing, smell, touch and taste.
An effective VR experience enables users to temporarily forget the ‘real’ world and instead feel a part of the virtual world they are experiencing. These ‘worlds’, also known as virtual environments (VE), can be anything from the corals of the Great Barrier Reef, to deep space, to inside cells, or to the land of the fairies.
While the idea of VR is not new, in the last five years great leaps in technology have seen it finally begin to break into the mainstream. This is partially due to the gaming and entertainment industry, which has driven much of the development, and the emerging interest in the technology by large multinational companies. Big bucks are being thrown at advancing the technology even further, with major technology companies working to become leaders in the field.
Achieving total immersion—that is, making the experience so real we forget that it isn’t—is the ultimate goal of many VR developers. There are still hurdles ahead, but we may not be as far away from the finish line as we think.
Head-mounted displays
Mention VR and most of us picture the traditional head-mounted displays (HMD), those cumbersome, futuristic-looking goggles that cover the eyes and part of the head.
Thankfully, modern HMDs are considerably less bulky, much lighter and far more sophisticated than their earlier counterparts.
With so many companies entering the VR market, there are a variety of different HMD options available, ranging in price and quality—from $15 for the most basic DIY models (literally a piece of foldable cardboard that turns your smartphone into an HMD), all the way up to $1,500 for the top-tier commercial models.
Seeing is believing
In terms of visual display, the current HMDs use the same key principle: stereoscopic display. Viewers look through biconvex lenses, at two screens (or, in the case of low-end models, one screen divided into two parts). Each screen displays a slightly different angle of the same scene. When viewed together, they merge perfectly (thanks brain!) to create a 3D image simulating depth. This is based on the same principles as 19th century stereoscopes, which allowed a viewer to observe a stationary image in 3D. The lenses also help to ‘wrap’ the image around your field of view, a bit like a fish-eye lens.
So if you can get a VR experience for $15, why spend more? Well, as with most things, you get what you pay for. The human brain is incredibly savvy and can quickly tell when something isn’t quite right. When this happens, we are essentially jolted out of the VR experience.
To begin with, we need a high screen resolution. In the more expensive VR models, two OLED screens, sitting only inches from the eyes, send out 2,160 x 1,200-pixel images, resulting in sharper, clearer visuals. The new technology can also accommodate higher refresh rates—that is, the number of frames displayed per second. A typical TV has a range of 60 hertz (hz), or frames per second (FPS). For VR, rates are up to 90 hz (FPS). The higher the refresh rate, the less likely the image is to ‘blur’ or ‘tear’ when you move your head, creating a smoother, more seamless visual experience.
Field of view (FOV) is another important part of ensuring the VR experience feels real. A person with healthy vision has around a 180-degree FOV when staring straight ahead. With eye movement, this increases to 270 degrees. At present, the high-end HMDs are capable of 110-degree FOV, which gives the user some peripheral vision. The cheaper cardboard units sit at 90 degrees.
Latency—the lag time between the information being sent to the device and the device responding—is also crucial to a successful VR experience. If you are in a virtual world and move your head to the left but the image takes several seconds to catch up, your sense of immersion will be immediately reduced. To convince our brains that we really are in a different world, latency must be less than 20 milliseconds. Anything more and our incredibly cluey brains begin to realise they’re being tricked. Latency can be affected by a variety of factors, including cables, screen quality, graphics processor and central processor capacity.
Tracking devices
In order to capture and respond to a wearer’s orientation, perspective and position a range of tracking devices are used, which communicate with the system’s central processing unit (CPU).
Tracking devices can be electromagnetic (orthogonal coils); inertial (a magnetometer, gyroscope or accelerometer); optical (cameras/LEDs); ultrasonic/acoustic (transmitters); or a hybrid of several trackers which accurately record and respond to movements across all three dimensions.
Basic head-tracking systems are able to track only the rotation of a user’s head, known as the pitch, yaw and roll. Internal components within the device such as an accelerometer, magnetometer and gyroscope give you a general sense of motion and orientation. They allow your device to capture your head movements and ensure the picture you’re seeing on the screen responds accordingly, but in many ways the experience is relatively ‘fixed’.
The more advanced units are equipped with position tracking. Position tracking registers the exact movements of a person’s head, so if you look up, down, backwards or forwards, the view will change to match your orientation. Even subtle movements, such as looking around a corner, can be detected.
Some higher end devices also come with external position trackers, which are placed on your desk or somewhere near where the system is used. These position trackers transmit additional information to allow the system to know where you are physically within the 3D space.
The drawback of position tracking devices is that they require the user to be attached via cables to the CPU, which can reduce mobility.
Audio
Seeing may be believing, but if you want a truly immersive VR experience, you need the right sound to accompany it. Imagine you’re seeing a gorgeous VR fireworks display—something so impressive it would put Sydney Harbour on New Year’s Eve to shame—but the sound is delayed by several seconds, is too close, is coming from the wrong direction, or is more like violins than explosions. Any of these will ensure your brain will refuse to accept the VR illusion.
This is because every sound in our environment has its own geospatial position and orientation, from the air conditioner humming above to the door closing behind you. Your ears are constantly processing a continuous stream of visual and non-visual cues to identify those sounds and map where they are coming from.
Developers have been working to come up with 3D audio solutions that mimic this natural listening experience. These build on earlier ‘surround sound’ technology found in many cinemas and homes.
Binaural recordings use two microphones, generally placed on a mannequin head where the ears would sit, to capture sound. The recordings result in distinct left and right sounds which, when played back, give the listener the sensation of being in a 3D environment.
Ambisonics is another recording method (originally developed in the 1970s) that uses a fixed-point spherical microphone. The advantage of ambisonics is that it can capture sounds from not only 360 degrees, but also above and below, resulting in a more authentic listening experience.
The introduction of head-related transfer function (HRTF) technology to the upper-end devices also improves the immersive experience by simulating the changes to a sound as it reaches you, creating realistic sounds in all directions. Various software programs can be matched with binaural and ambisonic recordings to further process and synthesise the sounds and use them in a variety of ways. These sounds can be matched to the visuals you’re seeing via the tracking devices to create a true 3D audio spatialisation.
Even monophonic sound sources can now be spatialised in 3D relative to the user’s head location via plug-ins for game engines. Spatialisation is the process of modifying sounds to make them localisable, so they seem to originate from distinct locations relative to the listener.
Interactivity
There are generally two types of VR experience—passive and interactive.
Passive experiences allow the user to be immersed in a VR world, but not to interact with or alter it in any way. For example, the simulation might be sitting in a submersible exploring the Great Barrier Reef. The user can look up, down and around but, while they can choose what to focus on, they cannot change the course of the submersible, stop it or, for example, poke a passing fish. The route is predetermined and every user has essentially the same experience. This is where some of the definitions of VR get confused. Michela Ledwidge, a technologist in Sydney, describes these encounters not as true ‘VR’, but instead as a 360-degree video playback experience.
As enjoyable as these experiences are, VR developers have found that it’s the interactivity of the virtual world, rather than its hyperrealistic appearance, that prompts a stronger immersive response in users. Being able to pick things up, open doors, change direction, feel sensory stimuli (such as heat, cold or wind) and have some autonomy over the experience keeps viewers more interested and engaged, constituting a more realistic VR experience.
But achieving a realistic level of interactivity is difficult, both in the creation of interactive content and in the development of haptic technology. So while developers are getting pretty good at fooling our eyes and ears, how are they faring with the other senses of touch, taste and smell?
In 2017 there are various devices on the market or in development that can allow users to manipulate their virtual environment.
Handheld controllers and gloves bring a level of interactivity by tracking the movements of users’ hands and allowing them to not only see their hands within the VR world, but also receive real-time feedback, such as vibrations from picking up or touching virtual objects. Foot-based controllers work in a similar way.
Taking interactivity even further, various developers are working on sensor-packed full bodysuits, which can allow the wearer to experience touch, vibration, temperature, shape, motion and texture within the virtual world. This is achieved through motion-tracking technology, small motors and sensors.
Building on HMDs, researchers at the University of Singapore have created accessories that can attach to a normal VR HMD to simulate taste (in particular sweet tastes) and atmospheric conditions, such as wind on your face or sun on your back.
While some of these technologies are still in development, or priced out of the reach of most consumers, they are exciting innovations that will continue to be improved.
The more parts of your body you can fool simultaneously, the closer you get to having those wonderful immersive experiences and feeling true presence.Chris Kohler, Wired Reviewer
Is VR safe?
When assessing any new technology, questions of safety will always arise. With VR, there are two elements of safety that need to be considered: physical and mental.
On the physical side, the main injuries caused by VR have been people ‘forgetting’ the real world around them and falling over, bumping into tables or walls or forgetting that virtual objects are not real and ‘falling’ through them. VR can also cause motion sickness, nausea, eyestrain, headaches and, in some people, seizures. While these symptoms generally disappear soon after the equipment is removed, or can be avoided by taking frequent breaks, some companies have put age restrictions (not recommended for children under 13) on their devices. There have yet to be any long-term studies on the physical effects of VR, but at present it is generally considered to be safe for adults.
A headache is one thing, but the other aspect of VR—how it affects your brain and mental health—is another. Preliminary studies have shown that VR experiences can remain with us as we re-enter the real world. This is because, although the simulation is fake, the emotions it can elicit (such as fear or happiness) are real. While this could have positive implications for the treatment of mental illness such as anxiety, depression and post-traumatic stress disorder, it also means that negative experiences, such as that experienced by Jordan Belamire, are not left behind in the virtual world. More studies are needed in this area.
Where to from here?
Going by the current interest in VR, it seems the time may have finally come for this technology (it only took 70 years!). While initially the realm of the gaming and entertainment industry, some of the world’s largest companies have now jumped on the VR bandwagon, and the potential applications of the technology are endless. Increasing competition and demand should continue to drive development, resulting in even more realistic worlds and experiences, while bringing the price down to a point that it is accessible for many consumers.
The current technology is impressive, but there is still a long way to go. Improved tracking technology, more user-friendly (and cost-effective) haptics and, importantly, creating virtual content for the devices, are essential if VR is to live up to the hype and become as real to us as our real lives.
Keep an eye out for another topic soon on how VR is being applied.