Mixed Reality and Guidelines for Developing in XR: Part 2

Part 2 of 3: The Cube Example

In part one of this series we touched briefly on what the XR landscape is and who the major players are in this space. Once you have an understanding of the general space, it is then important to grasp just how fast this space is changing with new innovations emerging all of the time. In this quickly evolving technological landscape we have a situation where the technology and applications are advancing to a degree where usability guidelines no longer apply. If you do not understand the factors of usability and user centered design in spatial computing applications then you are not going to be able to create intuitive interfaces with natural affordances allowing the user to easily navigate virtual information living in the physical world.

Let’s focus for a moment on a much simpler task to highlight what this complex ecosystem might look like. Let’s say that you want to create a virtual cube that can be placed on top of a physical table that you then knock off and watch fall to the ground and bounce. To achieve a realistic experience requires many capabilities such as dynamic mapping, depth-sensing, spatial anchoring, light estimation, and shadowing, just to name a few. The user must also be able to position the virtual cube perfectly on top of the physical table and have this appear continuous over time and across multiple viewing perspectives (Arifin et al., 2018). To pull this off requires an understanding of the user’s position in the world, how they perceive the virtual cube in relation to the real-world (e.g., believability, object permanence) and how the user understands the affordances of the tools that bridge the virtual to the physical.

Screen Shot 2020-02-26 at 2.05.09 PM Image: AR Virtual Cube on a Real Desk

The physical world contains many more cues for understanding the constraints and affordances for actions that cannot currently be represented accurately in a computer simulation (Bowman, 2014). In many cases, new XR applications can be quite difficult to navigate for those unfamiliar with the technology. In the example of the virtual cube, the user must be able to create and interact with digital information that in turn must interact with the physical world. While this can be achieved to some extent using hand or voice gestures or with a hand-held controller, any additional interface creates an added complication that can detract from the realism of the experience.

In real life, the cube task would not require a person to seek additional cues in order to push it off the table. One would simply mosey on over to the cube, extend a hand, and use their fingers to apply force until gravity took its course. Even in the case when natural hand gestures, the user still would not be able to physically move their hand in the same fashion across the virtual cube seeing as it is just an array of light particles… waves, regardless, it lacks a tangible mass. Anyone who has seen the movie Ghost with Patrick Swayze knows just how difficult it is to move an object when you lack a physical body. Screen Shot 2020-02-28 at 11.08.36 AM Image: Ghost, a 1990 romantic fantasy thriller film starring Patrick Swayze, Demi Moore, & Whoopi Goldberg.

The benefits of MR will only be achieved if the user interface (UI) is actively managed as to maximize the relevance and minimize the conditions of the virtual material relative to the real world. Moreover, the level of UX needs to also take into account the degree of user knowledge. This is especially true with applications using new technologies which are notorious for providing inadequate UX to users with little to no prior knowledge (Arifin et al., 2018).

There remains, to date, a lack of research or standardized metrics for understanding the user experience in XR or in how to go about evaluating that experience. Recent research is starting to pave the way to uncovering new and better ways to design XR experiences. Endsley and colleagues, for example, have developed a set of AR design Heuristics by leveraging affinity diagramming, expert evaluations, feedback from MR designers, and good old statistical analysis (2017). While I touch on some of their Nine AR Heuristics in this article, I do recommend reading Augmented Reality Design and Heuristics: Designing Dynamic Interactions for a more detailed account.

Check out Parts 1 and 3 of this series:

Part 1: MR and Guidelines for Developing in XR: Part 1 - An Introduction to XR

Part 3: MR and Guidelines for Developing in XR: Part 3 - Creating an Mixed Reality Experience