News / 26.09.2018
Tips For Producing Audio Content For VR Applications
Mark Jones, Audio Engineer, REWIND
As with most applications of interactive audio, the objective of sound in virtual reality is to make the experience as immersive as possible through replicating reality. Though stylistic aspects may inform the actual sound design (for example, you are unlikely to use sci-fi-style sounds in a fantasy setting) the behaviour of a sound in the virtual environment must reflect the behaviour of sound in the real world.
Here are my five tips for producing audio content for VR:
TIP 1: Use sound to expand the environment
Most mainstream VR headsets have a very limited field of view, creating a kind of “tunnel vision”, but this can be compensated for by using sound to “expand” the environment beyond what can be seen. This can be accomplished by using multiple sounds at a variety of distances from the player, in any position, and applying the following techniques:
– Separating very distant sounds – such as city traffic or music from a nightclub – to cover the full 360 degrees around the vertical axis. This will help to “fill” the environment.
– Placing nearby subtle ambient sounds just out of sight, such as the dull rumble of machinery on the other side of a wall.
– Placing “exposed” ambient sounds on visible objects, such as static from an untuned radio. These will always be the most prominent ambient sounds, but should not be overly distracting.
TIP 2: Use sound to draw the player’s attention
With a limited visual field, audio plays an enhanced role in allowing the player to locate objects they can’t see directly in front of them. Sound can be used as a powerful ‘attractor,’ grabbing the player’s attention and drawing them towards a certain object or location. For example:
– Loud sounds can prompt the player to immediately look in the direction of the sound, it is not necessary to deafen the player with something that is obnoxiously loud, the sound merely has to stand out.
– Continuous sounds can be used to remind the player of the presence of an object.
– Sounds with exaggerated volume attenuation and doppler effects can emphasise the impression that the object is moving. For example, a vehicle driving towards the player.
TIP 3: Be aware that the player is the centre of the experience
In a typical game displayed on a 2D screen, the in-game player character is not in the same spatial location as the player. In VR, this is not the case, as the player is always in the centre of the experience, and the sound moves around them. Using knowledge of acoustics, we can deliver a convincing, realistic experience by using three core components:
– Realistic attenuation (the drop in volume of a sound over distance). This is not a linear relationship, and most audio engines will include a realistic attenuation curve that is used as a default.
– Binaural spatialisation (a system that simulates the physical properties of a human head in the game’s 3D environment). In the real world, a noise behind a person will sound different from an identical noise in front, binaural spatialisation simulates these properties of human hearing in the virtual environment.
– Reverberation (echoes of a sound as it reflects from the surfaces in a space). A good reverberation simulator will take into consideration the materials these surfaces are made from. For example, a room built from solid stone will echo more than a room covered in fabric.
The major game engines such as Unreal and Unity are typically capable of natively supporting these core features without additional plugins being required, though audio plugins such as Steam Audio or Google Resonance Audio can be used to provide additional functionality such as reflection-based reverberation simulation.
TIP 4: Reinforce a sense of 3D space by emphasising spatial sounds
In order to help reinforce the feeling that the player is in an expansive 3D environment, the spatial nature of sounds can be emphasised through several means:
– Separation: Physically separate similar sounds by placing them beyond the audible range of each other. If there are two objects in the same room that should be producing similar sounds, create distinct variations of the sound. If two sources are producing the same sound (especially if they are widely-separated), the player will lose the ability to precisely locate that sound.
– Isolation: Position dissimilar sounds in proximity, preferably with different key frequencies. This will give each sound enough “acoustic room” in which to be heard.
– Verticality: Binaural audio allows the player to more easily locate sounds in the vertical axis, and as such, it is advisable to position sounds above and below the player. Otherwise, the lack of vertical separation between sounds will give the impression that the player is on an isolated 2D plane.
TIP 5: Ambisonics can be used for background ambiences
In some cases, there may be a need to limit the number of discrete sounds being used in a scene. In this case, ambisonic audio (a multi-channel audio format that changes to reflect the direction the player is facing) can be used. Here are three useful points to consider:
– Pre-rendered background ambient loops can be used to fill out a space, as ambisonic audio files can combine an unlimited number of discrete audio sources in a 3D space into a single file. This approach is best used in situations in which the player is unable to move, as the ambisonic audio cannot compensate for changes in the player’s location relative to the “location” of the audio.
– Spatialisation is not as precise as a mono audio source, so ambisonics aren’t ideal for sounds close to the listener. These sounds could be split from the ambisonic mix and played back separately.
– As ambisonic audio is pre-rendered, it is particularly useful for 360-degree videos or other non-interactive VR content. Ambisonic audio can theoretically combine every sound in an entire experience into a single file – useful for linear media on platforms with limited performance, such as mobile phones.
A final point to be made is that it is important to trust your instincts over any set of rules: if something feels wrong, then it probably is. In one of my projects, I had a sound attached to an object that the player could throw. A realistic attenuation just didn’t feel right, so I made it into an unrealistic linear attenuation, and fixed the issue. This is a creative process that is informed by a knowledge of acoustics; not dictated by it!
Corporations such as Walmart, Renault Trucks, and BAE Systems are employing immersive technologies to drive efficiencies, and these companies are seeing a strong return on investment.
The original Oculus Development Kit (the DK1) got me into VR. We built our own room scale VR using a DK1, three XBOX 360 Kinects, three tracking laptops, one backpack laptop, and a Razer Hydra.
I first met former broadcaster, Sir Martyn Lewis, back in January. We met at an event I was hosting on marketing innovations in London.
Autism is what is known as a spectrum condition, meaning it affects individuals in different ways. Adults and children with autism have challenges in behavior, social skills, verbal and non-verbal communication, as well as sensory and attention issues that impact their lives.
At South by Southwest in 2018, Bose introduced its AR glasses to showcase what sound-based AR might look and feel like. Combining data from embedded motion sensors with GPS information from your phone, the glasses knew where you were and which direction you were looking and moving. Small speakers fed sound to the wearer’s ears.
Throughout time, humans have used visual-based methods to help them remember information – from early cave drawings to modern day video. VR is not only the next logical step but a step in the right direction: research suggests we retain more information and can better apply what we have learned after participating in virtual reality exercises.
Virtual reality can change what a person sees, how they think, what they feel and even how they behave. This is because users believe they are present in the virtual environment they find themselves in.
HP’s new virtual reality headset, ‘Reverb – Professional Edition’, launched yesterday, joining HP’s comprehensive commercial VR portfolio. The headset has a staggering 2160 x 2160 panel per eye and 114-degree field of view.
SWSW 2019 was a whirlwind of enlightening talks, networking, tech innovations, and Texas BBQ! While we are still on a SXSW high, we wanted to share the five things that stood out for us at this year’s event…