Wizard of Oz Testing: Validate Complex Interactions Without Writing Code!

In the UX world, there are several methods/techniques that can be used to answer research questions. According to chapter 5 of Understanding Your Users, user research can be done with any type of timeframe and budget. While some methods can take a long time such as A/B testing, there are others that can be completed in a matter of a few hours. For this blog post, I wanted to focus on the Wizard of Oz (not the movie, but the UX method).

What is the Wizard of Oz Method?

Sara Paul and Maria Rosala explain this type of user-research method perfectly: “The Wizard of Oz is a user-research method where a user interacts with a mock interface controlled, to some degree, by a person.” Imagine, as an example, that your team was tasked with testing a new voice assistant. In this case, the research team is looking to understand how users would interact with the AI. Even though the AI isn’t created yet, the team created a physical cardboard mockup that the user would talk to with someone on the other end speaking through a bluetooth speaker. Similar to the movie, users would believe that who they were talking to was autonomous even though they were actually controlled by a hidden human operator. 

Instructions and Tools Needed to Perform the Wizard of Oz Experiment

To accurately execute a WOZ study, it’s important to create a flow of how the system should behave. Then, a minimum of two researches are needed as one needs to act as the moderator to guide the participant while the other is the wizard who is secretly connecting the user’s wants to the system. 

The tools needed are extremely varied in these experiments but some common ones are a screen or communication tool. 

Research Studies

1. Gesture Control Testing (2018)
   • Nicole McGovern uses the WOZ method to test their “gesture recognition platform” for controlling YouTube videos. The setup looked like they had created a sensor that could track the participants movements and gestures in real time. However, the device was actually did not function at all. Instead, there was someone in the room who was pretending to take notes but actually had a wireless keyboard that could trigger the various shortcuts that the subject performed. As a result, it made it seem like the participant could control the screen with only her hand gestures! McGovern noted in her blog that the participant had no idea and was completely fooled. The team learned that the prototype might have been a little too accurate and did not account for potential glitches that might occur if they were to use the real product. Even though their experiment worked too perfectly, they needed the prototype to feel realistic and not just functional. 

1. Virtual Health Coach (2021)
   • In this study, Andrea Seet prepared a pretend “chatbot” that participants had to message to in order to figure which pathways the chatbot could have conversations with the participant towards. Although the chatbot appeared to be autonomous, it was actually Andrea on the other end who was responding. Through this experiment, the team was able to validate the conversation follow and develop pathways (Social Support, Health Improvement, Self-motivated, Less Motivated) before a single line of code was written.
     

The Wizard of Oz method is a perfect tool to figure out what sort of potential results one might receive if the finalized product were actually ready to be of use, without investing too much resources into a product that may/may not produce the desired results. Although there is a little bit of trickery involved, this setup can product high-fidelity insights if one is on a tight budget.