Defining Reality: A Look Into What XR Experience is Best for You

So you have been thinking about diving into XR (eXtended Reality) but you are unsure where to even start or what all these emerging technology terms and acronyms mean? To help you get your heads around the various concepts and to explore what type of application would best fit your project’s needs, we created a handy guide.

360 Video

360 Video (also known as spherical or immersive videos) captures a panoramic view of an environment. Many associate 360 Videos with live action but they can also be animated or even a mixture of the two.

Interactions:
360 Videos have more limited capabilities and are not always interatable. Those that are interactable employ user gaze based control or a controller with simple UI elements.

Examples:
You can find many 360 Videos on platforms like Youtube and Vimeo. You can watch them on your desktop, your phone or tablet, and/or cardboard. Production companies like to start with 360 Videos to dip their feet into XR experiences; however, this does not mean your experience needs to be simplistic. Storytelling grows as you branch together 360 Videos like in Google’s The Hidden Worlds of National Parks.

Use Case Recommendation:
360 Videos are great if you want a more linear story or user journey. While you will still need to teach your users how your 360 Video works, there is more familiarity with traditional mediums. Additionally, if you need to cast a wide net then 360 Videos are one of the most accessible options for users to experience your content.

Handheld Augmented Reality (AR)

Handheld AR overlays the digital on top of the physical world. As the name suggests, it fits in the user’s hand, usually on a smartphone. These experiences benefit from its portability and ubiquity of mobile devices.

Interactions:
While Handheld AR can present the user with a range of interactions, it is most effective when the user wants to add elements into the environment and/or create illusions. Think of it as an instant Photoshop that uses facial or positional geometry.

Examples:
Companies are already engaging the wider population with applications such as Pokemon Go, Instagram & Snapchat and the upcoming Minecraft Earth. The use of AR within these applications are fun, brief and social. Filters that superimpose onto faces are the biggest trend; in particular, we’ve seen L’Oréal heavily invest in this technology for their ‘try before you buy’ applications.

Use Case Recommendation:
As mentioned earlier, Handheld AR is best exemplified with brief, social or marketing / promotional experiences. If your application’s core loop rests on your user’s engagement with a social network or enhances the user’s relationship with the company, brand or IP, then this technology might be right for you. We recommend keeping the interactions simple.

Dedicated Augmented Reality (AR)

Dedicated AR is a device that attempts to transform the physical world by utilizing the digital. By using a specialized, wearable device, such as a head-mounted display, the user engages with both the digital and the physical at the same time to enhance their experience.

Interactions:
While you’ll want to match your interactions to your application, a broad example of a few types of interactions you’ll find include voice commands, controller schemes on the actual devices and gesture-based interactions when looking into Dedicated AR.

Examples:
There are a range of wearable devices like Microsoft's Hololens 2 which is great for complex hand gestures and a convincing interface, Magic Leap One with a more accessible price point, and Google Glass Enterprise Edition which is aimed at business / enterprise users. As with much of XR technology, many associate AR with the visual; however, we think it is important to keep in mind that AR can extend into the auditory. BOSE AR, for example, uses motion sensors to detect head and body movements and exemplifies how sound can enhance the environment.

Use Case Recommendation:
If you want to improve upon your current workflow, allow for better project management and collaboration or generally improve everyday functionalities, then we’d recommend a Dedicated AR experience. Basically, this technology is great if your users can benefit from digital information but still need to interact with the physical world. When designing we’d recommend taking advantage of naturalistic interfaces while also employing traditional 2D ones that users already have built-in affordances.

Mixed Reality (MR)

Mixed Reality (MR) is a type of XR experience that merges the physical and virtual worlds. You can manipulate, interact with and/or be affected by items in both physical and virtual environments. You might be asking at this point, how is this different than AR? Here we’d like to note that there are many different definitions for MR, as explained in Speicher’s What is Mixed Reality. While many will associate it with Milgram's continuum, you will also hear the term come up depending on how a brand wants to market their experience or device. As to how we will define it here, we want to note that rather than merging the physical and virtual worlds like with AR, MR ramps up the virtual with the use of props and additional peripheral devices. Because headset such as Microsoft's Hololens 2 and Magic Leap One support interaction with virtual objects that are anchored in the real world, they are also considered MR devices (as well as the mentioned above Dedicated AR). What we’d like to make clear is that MR experiences can use either AR or VR devices in combination with the physical world.

Interactions:
In addition to 6DoF and the large variety of possible interactions, MR benefits from tactical and other sensory perceptions in the environment. Similar to experiences that use VR to enhance immersion, MR can include anything from olfaction, breathe, changes in temperature, gusts of wind, custom devices such as treadmills or haptics, and one-to-one experiential set or prop design.

Examples:
A popular MR experience you may have heard of is Birdly, an ultimate flying simulator that merges VR with robotics to create a full-body in-flight experience. Additionally, we are continuing to see content creators experiment with MR in festivals like Sundance’s New Frontier which showcased Breathe, a Magic Leap experience mixed with biometric sensors. We’ve even seen MR used for promotional campaigns with the partnership of The Void, Madame Tussauds and Sony for Ghostbusters: Dimension.

Use Case Recommendation:
Though MR experiences can help users feel highly immersed, the resourcing and installation costs can be hefty. MR can be seen as a heavy lift and usually requires additional investments to pull off successfully. Identify whether the users would benefit from these multi-sensory approaches. Certain health-care based or complex simulators (eg: cognitive-behavioral therapies, Alzheimer’s therapy, flight simulations) that rely on cross-modal perceptions are some good examples. Additionally, events that have exposure to a wider audience, location based installations, and other marketing & branding experiences work well in MR.

Mobile VR

Mobile VR basically uses a set of lenses, light-weight cushioning and a smartphone to create a sense of depth for the user to experience VR content. While the price is definitely affordable and the tetherless option is great, don’t hold your breath for mobile VR’s reach as an instant household classic. The graphics level and run-times leave something to be desired. Essentially, your graphics and performance are only as good as the phone’s.

Interactions:
Mobile VR is basically a step up from 360 Videos. Most headsets have at least one controller and can provide users with 3DoF (3 Degrees of Freedom). This means you can track forward and backwards movements but not position. This is rotational tracking - you can look up, down, side to side or title your head but if you lean or move your head’s position than the entire virtual world moves as well.

Examples:
You can easily find experiences on Samsung Gear, Oculus Go, Google Daydream and, while not exactly Mobile VR, you can transform your Switch using Nintendo’s Labo VR Kit (let’s face it - those mini-games are extraordinarily fun!). Experience you’ll find on their application stores can range from educational to narrative and gaming.

Use Case Recommendation:
While it may be phasing out and you’ll want to avoid developing on it for a wider audience, Mobile VR can make an easy, cost-efficient, internal tool. Mobile VR is great to simulate some light training or showcase a bare-bones prototype. Basically, if your users need to be in a different environment, you want more interactivity than a 360 Video, and you don’t want to dip your toes into higher-end experiences just yet, then Mobile VR is right for you.

Standing / Room-scale VR

Standing / Room-scale VR is where we dive into movements and positional tracking with full 6DoF (6 Degrees of Freedom). Users can move around seamlessly as what you do in the physical world is mapped in the virtual one. Additionally, Standing / Room-scale VR are often associated with professional level graphics quality and controllers with haptic feedback.

Interactions:
There is a wide range of interactions available when diving into Standing / Room-scale VR including gestures, gaze and voice. We also see many types of controller-based interactions because Standing / Room-scale VR is popular within the Gaming industry. However, no matter the type of project you are producing, it is important to choose interactions that work best for your users without adding too much cognitive load or presenting an overcomplicated controller schema.

Examples:
Oculus and HTC / Valve are the biggest competitors in this category. Oculus and HTC / Valve have been racing each other for tetherless headsets (Oculus Quest vs VIVE Pro) and sensor-based controllers (Oculus Quest Hand Tracking vs Valve Knuckles). As for content, you may have heard of Beat Saber, the most profitable VR application selling over 1 million copies in a year and making 20 million in revenue. We’d also recommend looking out for Half-Life: Alyx, one of the most anticipated games of the year.

Use Case Recommendation:
Consider whether it is important for you to completely simulate a digital environment and the types of interactions your users would need to perform for the experience to be most beneficial to them. While entertainment, healthcare and construction / manufacturing are just a few industries that have incorporated Room / Standing VR experiences, we are seeing a lot of traction from companies for looking into training simulations. These simulations can be anything from educating employees on customer service to health & safety protocols to specialized medical retention platforms. If you are interested in VR training simulations, we’d recommend checking out this KLI article for information on how to Get Started with Training in VR.