Strategies for Usability Testing with VR

Interview by Samantha Silver of Rick Damaso, the Lead Researcher & Strategist for Key Lime Interactive.

Strategies-for-Usability-Testing-with-VR-img1

For those that have never run their own VR study, what are some of the biggest concerns that may be different from testing traditional screen based apps?

Seeing the growth in VR and its applicability beyond the gaming world is incredibly exciting. We’re now seeing VR applied in fields like education, healthcare, and consumer retail. However, testing in this new environment presents new challenges and considerations that many UX researchers have never had to consider.

Perhaps the most basic, yet significant of which is testing for Simulation Sickness.

For those who are not familiar with Simulation sickness, think of motion sickness one would normally experience during a car ride or a plane ride, but now occurring within a simulation. For example, “I know my feet are not moving, but I can see that I am traveling at a high rate of speed in this simulation.” As you can imagine, this can be pretty jarring, even for advanced users.

That being said, my experience has indicated that people have very different thresholds for what would cause them to experience simulation sickness. For example, avid gamers who have spent tons of time in First Person Shooters have become accustomed to the approximately 100-degree field of horizontal view. For reference on how this can impact an individual’s capacity to get simulation sickness, the human eye is capable of a view of 180 degrees horizontally and around 130 degrees vertically. VR is similar to the First Person Shooter games in the sense that a user is restricted to a much narrower field of view while wearing a head-mounted display. This alone may give users the feeling of being in an enclosed environment and lead to claustrophobia or nausea — something we typically never consider when testing in traditional screen based scenarios.

So with that being said, do you have any advice for designers who are thinking about creating a new VR or Immersive Video experience?

Yea, definitely. Google and Samsung have taken a big step towards putting VR in the hands of the masses with the Cardboard and Gear. What used to cost several hundred, if not thousands, of dollars to set up in your home, you can now experience using your iPhone or Galaxy S7 and a $20 piece of hardware (cardboard).

Additionally, a new wave of immersive experiences like 360 videos is popping up everywhere. If you, or your organization, is considering making your own videos, there are definitely some important things you should consider when incorporating these kinds of tools into your testing.

Let’s take 360 videos as an example: As you’re recording the video, treat the camera like your end user — That means keeping your character, or stimulus within eye level of the camera and at a reasonable distance. To help do this, ask yourself, how close would you want to be from someone you are having a casual conversation with? Users can rotate their field of view almost 360 degrees, but can not get closer or further in 360 videos. Forcing users to strain in order to see can lead to eye strain and a poor user experience. Another important thing to remember is do not rotate the user’s field of view. Remember, viewers will be stationary as they watch the video. The one control you have given them in the experience is where they can look, taking that back and rotating the fielding of view from portrait to landscape is the same thing as having someone flip your TV as your watching a movie. And while high-speed chases are cool, in 2D scenarios - the fastest way to get people feeling queasy is by placing them in a scenario where their brain tells them they’re moving but their body knows they’re standing still. That’s not to say you can’t have some movement in your experience, but I would recommend keeping the high-speed chase pursuits to a minimum.

What are some strategies that you have found to be essential when testing in VR?

Start small, very small. A good researcher knows how to design a UX study to isolate specific variables and deliver targeted feedback. However, the reality is that testing in VR is as much of an art as it is a science. This is especially true if your team is starting from scratch with a new experience or if your target audience may not fit the traditional VR demographic.

A couple of things I would consider is don’t forget to test how to test - we are all creatures of habit. Testing in a new environment like VR has challenged me to re-acclimate myself to different testing methods. If you’ve been running traditional usability tests and haven’t had an opportunity to conduct ethnographic evaluations in while, take some time to brush up with several pilot sessions before jumping into the real thing.

And also remember to evaluate your procedures and outputs - you’re not in Kansas anymore! There are several testing facilities that support full-fledged VR rooms. However, due to your personal accessibility to these facilities, most of the testing may have to take place in your own lab or design space. If that’s the case, setting up cameras in multiple vantage points and syncing it with the user’s viewing experience make analysis and reporting much more actionable.

So, what does this mean for the future of usability testing? What do the future uses of VR look like?

It is unclear what this means for the future of VR. Being a super nerd makes me a little biased when it comes to what I hope the future brings. The reality is that many still see VR as a technology that’s in a nascent state. Whether or not it is able to fulfill the promises of teleporting users into new realities while providing a full sense of agency and presence is still TBD. However, VR still has a long way to go.