News / 26.09.2018
Tips For Producing Audio Content For VR Applications
Mark Jones, Audio Engineer, REWIND
As with most applications of interactive audio, the objective of sound in virtual reality is to make the experience as immersive as possible through replicating reality. Though stylistic aspects may inform the actual sound design (for example, you are unlikely to use sci-fi-style sounds in a fantasy setting) the behaviour of a sound in the virtual environment must reflect the behaviour of sound in the real world.
Here are my five tips for producing audio content for VR:
TIP 1: Use sound to expand the environment
Most mainstream VR headsets have a very limited field of view, creating a kind of “tunnel vision”, but this can be compensated for by using sound to “expand” the environment beyond what can be seen. This can be accomplished by using multiple sounds at a variety of distances from the player, in any position, and applying the following techniques:
– Separating very distant sounds – such as city traffic or music from a nightclub – to cover the full 360 degrees around the vertical axis. This will help to “fill” the environment.
– Placing nearby subtle ambient sounds just out of sight, such as the dull rumble of machinery on the other side of a wall.
– Placing “exposed” ambient sounds on visible objects, such as static from an untuned radio. These will always be the most prominent ambient sounds, but should not be overly distracting.
TIP 2: Use sound to draw the player’s attention
With a limited visual field, audio plays an enhanced role in allowing the player to locate objects they can’t see directly in front of them. Sound can be used as a powerful ‘attractor,’ grabbing the player’s attention and drawing them towards a certain object or location. For example:
– Loud sounds can prompt the player to immediately look in the direction of the sound, it is not necessary to deafen the player with something that is obnoxiously loud, the sound merely has to stand out.
– Continuous sounds can be used to remind the player of the presence of an object.
– Sounds with exaggerated volume attenuation and doppler effects can emphasise the impression that the object is moving. For example, a vehicle driving towards the player.
TIP 3: Be aware that the player is the centre of the experience
In a typical game displayed on a 2D screen, the in-game player character is not in the same spatial location as the player. In VR, this is not the case, as the player is always in the centre of the experience, and the sound moves around them. Using knowledge of acoustics, we can deliver a convincing, realistic experience by using three core components:
– Realistic attenuation (the drop in volume of a sound over distance). This is not a linear relationship, and most audio engines will include a realistic attenuation curve that is used as a default.
– Binaural spatialisation (a system that simulates the physical properties of a human head in the game’s 3D environment). In the real world, a noise behind a person will sound different from an identical noise in front, binaural spatialisation simulates these properties of human hearing in the virtual environment.
– Reverberation (echoes of a sound as it reflects from the surfaces in a space). A good reverberation simulator will take into consideration the materials these surfaces are made from. For example, a room built from solid stone will echo more than a room covered in fabric.
The major game engines such as Unreal and Unity are typically capable of natively supporting these core features without additional plugins being required, though audio plugins such as Steam Audio or Google Resonance Audio can be used to provide additional functionality such as reflection-based reverberation simulation.
TIP 4: Reinforce a sense of 3D space by emphasising spatial sounds
In order to help reinforce the feeling that the player is in an expansive 3D environment, the spatial nature of sounds can be emphasised through several means:
– Separation: Physically separate similar sounds by placing them beyond the audible range of each other. If there are two objects in the same room that should be producing similar sounds, create distinct variations of the sound. If two sources are producing the same sound (especially if they are widely-separated), the player will lose the ability to precisely locate that sound.
– Isolation: Position dissimilar sounds in proximity, preferably with different key frequencies. This will give each sound enough “acoustic room” in which to be heard.
– Verticality: Binaural audio allows the player to more easily locate sounds in the vertical axis, and as such, it is advisable to position sounds above and below the player. Otherwise, the lack of vertical separation between sounds will give the impression that the player is on an isolated 2D plane.
TIP 5: Ambisonics can be used for background ambiences
In some cases, there may be a need to limit the number of discrete sounds being used in a scene. In this case, ambisonic audio (a multi-channel audio format that changes to reflect the direction the player is facing) can be used. Here are three useful points to consider:
– Pre-rendered background ambient loops can be used to fill out a space, as ambisonic audio files can combine an unlimited number of discrete audio sources in a 3D space into a single file. This approach is best used in situations in which the player is unable to move, as the ambisonic audio cannot compensate for changes in the player’s location relative to the “location” of the audio.
– Spatialisation is not as precise as a mono audio source, so ambisonics aren’t ideal for sounds close to the listener. These sounds could be split from the ambisonic mix and played back separately.
– As ambisonic audio is pre-rendered, it is particularly useful for 360-degree videos or other non-interactive VR content. Ambisonic audio can theoretically combine every sound in an entire experience into a single file – useful for linear media on platforms with limited performance, such as mobile phones.
A final point to be made is that it is important to trust your instincts over any set of rules: if something feels wrong, then it probably is. In one of my projects, I had a sound attached to an object that the player could throw. A realistic attenuation just didn’t feel right, so I made it into an unrealistic linear attenuation, and fixed the issue. This is a creative process that is informed by a knowledge of acoustics; not dictated by it!
Welcome to the Monthly REWIND for October
In this post, we’re going to go a little deeper into why the Quest will be an important milestone in the development of consumer VR.
The Oculus Quest is a device that not only delivers on technology, but content too, and at a price that won’t put off the masses.
Creating top quality 360º video requires more than just the press of a button…
While I take a moment to bask in our win, here’s a look back on what we’ve achieved…
Oculus’ Santa Cruz could present something of a ‘Nintendo Wii Moment’ for the VR industry – the device that propels the hardware into the mainstream through a focus on form, function, and simplicity over hardware power.
Welcome to The Monthly REWIND for September
Is developing for a single VR platform a better idea? Let’s look at the various pros and cons.
While some stats paint a picture that it’s all doom and gloom, there are plenty of alternative views and evidence that VR is ramping up.