• Skip to main site navigation
• Skip to quick links
• Skip to main content

• Home
• About us
• ICT accessibility
• Research resources
• Project tools
• News and events

Wizard of Oz

The Wizard of Oz method of evaluation is a specific type of observational evaluation, known as interactive observation. It involves a user interacting with the system under evaluation - normally a software based system or prototype - whilst a hidden operator processes input from the user and simulates the system output accordingly. During the evaluation, users are led to believe that they are interacting with and directly controlling the system.

When can the Wizard of Oz method be used?

The Wizard of Oz method provides a means of evaluating software-system prototypes without the need for full implementation. As such, the method is particularly useful if a system is expensive or demanding to implement. The method is also useful early in the prototyping phase, as a means of gathering user opinions without having to develop a fully functioning prototype.

How to conduct a Wizard of Oz evaluation

A Wizard of Oz evaluation requires two evaluators, plus the user. One evaluator acts to control the system under evaluation, whilst the other evaluator monitors the users actions. The RESPECT European Usability Support Centre project, lists 13 general steps to conducting a Wizard of Oz evaluation. These are:

1. Develop the wizard of Oz prototype, design the tasks, and recruit the users;
2. Allocate the role of wizard and the role of facilitator to the relevant staff;
3. Assemble the necessary equipment and inter-connecting software;
4. Select appropriate users to test the prototype, try to cover the range of users within the target population;
5. Prepare realistic task scenarios for the evaluation;
6. Pilot the evaluation procedure and ensure the wizard is well practiced in playing the role of the computer;
7. Ensure recording facilities are available and functioning;
8. Conduct each session. The facilitator instructs the user to work through the allocated tasks interacting and responding to the system as appropriate;
9. Conduct post-session interviews with the users, drawing upon pre-set questions and issues raised during the use of the prototype;
10. Debrief the users and thank them for their co-operation;
11. Analyse information obtained, summarise observations and user evaluations. Consider the themes and severity of the problems identified;
12. Summarise design implications and recommendations for improvements and feed back to design team. Video recordings can support this;
13. Where necessary refine the prototype and repeat the above process.

Advantages & disadvantages of the Wizard of Oz evaluation technique

Advantages

• Allows the system to be evaluated at an early stage in the design process;
• Provides a unique insight into the user's actions, gained from 'interacting' with the user during the evaluation.

Disadvantages

• Evaluators need intimate knowledge of the operation of the system in order to ensure a convincing representation;
• Difficult to evaluate systems with a large graphical interface element;
• The evaluation is reliant on a relatively large resource element (e.g. the system under evaluation, plus a system to control the evaluation; minimum of two evaluators).

Acknowledgements

• The Department of Trade and Industry. (1990) A Guide to Usability. Exeter, UK: BPCC Wheatons Ltd.
• RESPECT European Usability Support Centres. (1997) Requirements Engineering and Specification in Telematics. Deliverable D3.2: Methods for User-Orientated Requirements Specification: Wizard of Oz Prototyping. [accessed 23/06/08].
• UsabilityNet (2006). Wizard of Oz . [accessed 23/06/08].