Heuristic evaluation

Heuristic evaluation is an usability inspection technique developed by Neilson and colleagues in the mid 1990s. The technique involves the assessment of a product or system by a small set of evaluators, who are guided by a set of usability principles, known as heuristics.

Evaluations are usually carried out by several evaluators, operating independently of one another. This significantly enhances the effectiveness of the technique, since different people normally identify different accessibility problems; furthermore, individual evaluators very rarely (if at all) identify all the accessibility issues with a system.

Common usability principles (heuristics)

A list of 10 general heuristics was proposed by Nielsen and colleagues in 1994, based on the analysis of 249 usability problems. These are:

Visibility of system status - the system should keep users informed about what is going on, through appropriate feedback within reasonable time;
Match between system and the real world - the system should speak the users' language, with words, phrases and concepts familiar to the user, rather than system-oriented terms;
User control and freedom - does the system provide ways of allowing users to easily escape from places they unexpectedly find themselves, by using clearly marked "emergency exits";
Consistency and standards - the system should not leave users wondering whether different words, situations, or actions mean the same thing;
Error prevention - choose a careful design which prevents a problem from occurring in the first place;
Recognition rather than recall - minimize the user's memory load by making objects, actions, and options visible. The user should not have to remember information from one part of the dialogue to another;
Flexibility and efficiency of use - accelerators - unseen by the novice user - may often speed up the interaction for the expert user. Allow users to tailor frequent actions;
Aesthetic and minimalist design - dialogues should not contain information which is irrelevant or rarely needed. Every extra unit of information in a dialogue competes with the relevant units of information and diminishes their relative visibility;
Help users recognize, diagnose, and recover from errors - error messages should be expressed in plain language (no codes), precisely indicate the problem, and constructively suggest a solution;
Help and documentation - where it is necessary to provide help and documentation, such information should be easy to search, focused on the user's task, list concrete steps to be carried out, and not be too large.

Since the publication of the original list by Neilsen, the creation of new products has led to other heuristics being developed, more suited to the product under evaluation. However, the principles outlined above continue to provide the building blocks upon which an evaluation protocol can be developed. Where new products are developed, designers and evaluators should develop their own heuristics, by tailoring the above principles, according to the design guidelines and requirements of the product.

Conducting a heuristic evaluation

Heuristic evaluations consist of three main stages:

Plan the evaluation

The first stage is to decide how you will test the system. Heuristic evaluations employ three main approaches: 1) develop a set of tasks and ask the evaluators to carry them out; 2) provide evaluators with the system goals and allow them to develop their own tasks; and 3) ask the evaluators to asses the dialogue elements.

Which method you choose depends on the time available, the evaluators and the maturity of the system under test. If the evaluation is for a functioning product or system, the evaluators should be given specific tasks upon which the evaluation is focused. However, if the product is early in development, providing evaluators with the system goals and allowing them to develop their own tasks is more appropriate.

Conduct the evaluation

Prior to beginning the evaluation it is important to brief the evaluators so that each person knows what to do, in terms of the approach, goals of the evaluation etc.

The heuristic evaluation itself is performed by having each individual evaluator inspect the interface alone, using the heuristics for guidance. The evaluators should be briefed on the heuristics against which the product will be assessed prior to the evaluation.

Each expert should make at least two passes through the system, the first to get a feel for the product, the second to focus on specific interface element of the product. Only after all evaluations have been completed should the evaluators be allowed to communicate and discuss their findings.

Debrief & reporting

Once the experts have completed their individual evaluations, they should come together to discuss their findings. Duplicates should be removed and similar issues combined. The findings should then be prioritised, highlighting each problems identified and potential solutions.

When should a heuristic evaluation be performed?

Heuristic evaluation can be used throughout the design lifecycle – at any point where it is desirable to evaluate the usability of a product or product component.

How many evaluators are needed for a heuristic evaluation?

Evidence suggests that three evaluators usually identify just over 50% of the total usability problems. Increasing the number of evaluators to five, increases the percentage of usability problems identified to approximately 75%. However, increasing the number beyond this levels results in a diminishing scale of return; doubling the number of evaluators to ten, increases the percentage of usability problems identified to approximately 90% (Nielsen, 1994)

The number of expert evaluators used will depend on time and resources. However, it is recommended that not less than three evaluators should be used.

Advantages & Disadvantages of Heuristic evaluations

Advantages

Relatively inexpensive compared to other evaluation methods;
Intuitive and easy to motivate potential evaluators to use the method;
Evaluators do not have to have formal usability training;
Can be used early in the development process;
Faster turnaround time than laboratory testing.

Disadvantages

Individual evaluators identify a relatively small number of usability issues. As such multiple evaluators are recommended which can increase the cost of the evaluation;
Heuristic evaluations may not identify as many usability issues as other usability engineering methods, for example, user testing;
Heuristic evaluation may identify more minor issues and fewer major issues than would be identified in a think-aloud user testing;
Heuristic reviews may not scale well for complex interfaces;
May not always readily suggest solutions for usability issues that are identified;
Can be biased by the preconceptions of the evaluators;
In heuristic evaluations, the evaluators only emulate the users – they are not the users themselves. Actual user feedback can only be obtained from laboratory testing, or by involving users in the heuristic evaluation;
Heuristic evaluations may be prone to reporting false alarms – problems that are reported that are not actual usability problems in application.

Return to expert evaluation methodologies page

References

The information contained on this page was taken from the following sources:

Dinano, N. (2001) Heuristic Evaluation - a Step By Step Guide. Sitepoint (n.d.) [accessed 05/06/08].
Molich, R. & Nielsen, J. (1990) Improving a human computer dialogue. Communications of the ACM. 33(3), 338-348.
Nielsen, J. (1994) Heuristic evaluation. In: Nielsen, J. & Mack, R.L. (eds). Usability inspection methods. New York: John Wiley & Sons.
Nielson, J. (1994) Enhancing the explanatory power of usability heuristics. In Adelson, B., Dumais, S. & Olson, J.S. (eds.) Proceedings of ACM CHI 94 Human Factors in Computing Systems Conference 1994, April 24-28, Boston, USA, 152-158.
Nielsen, J. & Molich, R. (1990) Heuristic evaluation of user interfaces. In Carrasco, J. & Whiteside, J. (eds.) Proceedings of the ACM CHI 90 Human Factors in Computing Systems Conference 1990, Seattle, USA, 249-256.
Preece, J., Rogers, Y. & Sharp H. (2002) Interaction design: beyond human-computer interaction. New York: John Wiley & Sons.
Usability Professionals Association (2005) The Usability Body of Knowledge. [accessed 09/06/08].
Useit.com (n.d.) Heuristic Evaluation. [accessed 05/06/08].