Part 3 of 3: Creating a Mixed Reality (MR) Experience
In the first part of this series we covered what XR is and what types of technologies fall along it's spectrum. We then talked in depth about what it would take to create a simple MR experience including the need to take into account the prior experiences of the user and their interaction with the virtual and physical space. Things can get complicated very quickly so it's important to really think about the type of experiences you are trying to provide the user. Once you have a good handle on that it's time to break that down step by step and then take those steps and dissect it even further until you have a very detailed outline of the all of the possible ways that a user will perceive and interact with the digital world that you've created.
Understanding the design space
The three-dimensional nature of XR experiences makes the evaluation of design and user experience challenging. In the example from before, the design of the virtual cube must be able to fit on the physical table in a way that allows the user to make sense of it. The cube must have visual assets and invite tangible, physics-based metaphors for interaction. As such, the virtual cube should behave in a way we would expect a physical cube would of similar size and texture. Consistent with this, the use of visualizations and metaphors should mirror the cognitive heuristics that you would expect a user to have given their previous experiences and the context of the situation.
Don’t forget to also think about the type of hardware you will be using to host your experience and to design around its capabilities and limitations. Once you have an understanding of the design space and the hardware that you will be using, it is now time to find development tools that will work best for your needs. The image below, for example, is just one potential path of development tools needed to create a 3D spatial computing experience. Once you understand the design space and the design tools the world is now your virtual oyster.
Image: mobile AR tool kit. credit, source Michael Cohen
Communicating the design
You need to be able to communicate to your user the affordances and capabilities in a way that meaningfully connects to the user. If I were to give you an MR headset and asked you to participate in my virtual cube experience, one of your first questions may be how to generate the virtual cube? For this example, let’s say that the cube is floating in space and your job is to figure out how to place the cube onto the table. Your natural inclination might be to make hand gestures for grabbing and moving the cube. This would be an example of high interaction fidelity or the degree to which our actions used for a task in the UI correspond to actions for that task in the real-world.
Image: Mixed Reality is preparing students for collaborative workforce
A high-fidelity experience is the most natural for the user but requires mapping the movement of the user’s hand and fingers exactly, requiring precise placement, grasping, and releasing. This tends to get complicated fast, especially once you start adding more virtual objects of varying size to the mix. Therefore, it may be easier to adopt a semi-natural metaphor such as a 3D cursor that will allow you, the user, to visually target the cube, grab it, and move it to the table; creating a continuum of realism. Conversely, a low-fidelity experience might look something like a user selecting the cube and action from a 3D menu.
Evaluating the user experience.
It is important to first understand the experience of the user and how they interact with the physical world. If we want to design the virtual cube to move using gestures, we first need to have some understanding of the user position and motions. Users come in all shapes and sizes with different cultural experiences and as such it is difficult to create a set of universal intuitive gestures that all users will naturally do. Given the time and resources, however, you could train a deep learning algorithm using thousands of examples of different people gesturing to grab a virtual cube. Alternatively, you could train the user on a select number of gesture motions. In other words, it is critical to have some understanding on how the user will interact with the space when creating your assets.
It is also important to develop a set of heuristics in which to evaluate your system. The classic UX evaluation heuristics, however, cannot directly be applied to current XR technologies and therefore more research and user experience testing is needed to inform the design of XR applications. In addition to Endsley’s Nine AR Design Heuristics, I also recommend Hollerer et al. (2001) which is a bit dated but provides a nice foundation for planning and execution.
Image: Mobile Augmented Reality Scale
Some components to keep in mind while planning your experience should include Information Filtering of the design space or selecting only the most relevant information to the user, the task, and the current context. Another component is the UI Component Design in which you must choose an appropriate format in which to present that information based on available resources and tracking capabilities. Finally, use View Management to ensure that virtual objects match appropriately in the field of view that they don’t occlude critical information in the environment that might create safety concurs. It could be dangerous, for example, if that virtual cube on the floor now occludes your dog and you accidentally step on him.
Image: The author’s dog Odin playing with the concept of reality.
These are just some examples of new research paradigms that are being developed to help understand how to take the user experience from 2D to 3D. There is still a lot of foundational work to be done and things can go from simply to complex very quickly, but don’t let this deter you. This is an exciting time for XR, we are witnessing the birth of a new technology that is going to change the way humans interact with machines. We are quite fortunate to be a part of this rare opportunity to be active explorers and designers in this new unexplored frontier. Who knows what new UIs will be birthed from a design space with almost infinite possibilities.
Check out Parts 1 and 2 of this series:
Part 1 - Mixed Reality and Guidelines for Developing in XR - An Introduction to XR
Part 2: Mixed Reality and Guidelines for Developing in XR- The Cube Example
READ MORE: AR, Art and Snapchat, Rise of AI and Zero UI, Biometrical Authorization and the Future, AR and Tourism
Comments
Add Comment