Cost 219ter

3.0 Methodology

3.1 Equipment

 

The equipment requirements of this evaluation can be split into hardware needs and software needs. 

3.1.1 Hardware

Nokia phones running the Series 60 Symbian software were the only mobile phones that could be used for this evaluation trial. These phones supported third party applications such as Talks which enabled Blind and Partially Sighted participants to access all areas of the mobile phone so that these users could fully contribute to this evaluation.  Six Nokia 6681 phones were used and five Nokia 6680 phones were used for this evaluation.  Specific phones were used for the Portuguese trials as the six Nokia 6681 handsets did not have the Portuguese language on and the Portuguese participants did not speak English.  Although two different model numbers were used, the phones were identical in design and software versions.  The only difference was that the Nokia 6680 was a 3G phone and the Nokia 6681 was not.  The six Nokia 6681 handsets were purchased by RNIB (UK) and the five Nokia 6680 handsets were borrowed from Vodaphone Portugal for a ten-day period.

3.1.2 Software

 

Each of the six Nokia 6681 phones needed to have the screen reader Talks installed on them so that blind and partially sighted participants could participate in this evaluation.  The screen reader used was supplied by Nuance (www.nuance.com). In addition to the screen reader, a screen magnifier was also supplied as part of the software package.

3.2 Procedure

 

This evaluation was broken down into three parts, an expert evaluation, an end user evaluation and finally a comparison stage which compared the results obtained in the expert evaluation and the end user evaluation. 

3.2.1 Expert Evaluation

 

This was performed before the end user evaluation. This evaluation consisted of two evaluators completing three sets of heuristics which covered the three aspects of the mobile phone.  The three aspects are hardware, software and services.  Each heuristic principle within each aspect had checkpoints which provided greater detail as to what should be evaluated.  The evaluators were experts in the heuristics and mobile phones completed the evaluation independently. 

The three sets of heuristics were completed with the expert going through them methodically. The results of each heuristic evaluation were discussed between the two evaluators once completed.  This was to check that their answers were consistent.  The heuristics are as follows:

3.2.1.1 Hardware Heuristics

• The product needs to be easy to orientate
• The printed visual information needs to be clear
• The non-printed visual information needs to be clear
• The buttons, switches, dials, and joysticks need to be easy to locate visually
• The buttons, switches, dials, and joysticks need to be easy to locate touch
• The connectors, sockets, battery, battery compartment/back cover, SIM card holder, memory card and memory card holders need to be easy to locate by sight
• The connectors, sockets, battery, battery compartment/back cover, SIM card holder, memory cards, memory card slot need to be easy to locate by touch
• The other tactile information needs to be clear (texture/size/differences)
• The audible information needs to be clear
• The battery, SIM card and memory card must be easy to use
• The product needs to be physically easy to use
• Compatibility
• Instructions
• Packaging

3.2.1.2 Software Heuristics

• Allow choice over the input and output method
• Optimise the input method
• Optimise the output method
• Allow personalisation of the input
• Allow personalisation of the output
• Allow personalisation of the alerts
• Promote flexibility
• Promote consistency
• Provide easy to understand menus
• Provide intuitive prompts and informative feedback
• Promote easy to use Softkeys
• Error prevention and error recovery
• Promote Easy Connectivity and Compatibility with other devices and/or software
• Aid Task Completion

 

3.2.1.3 Service Heuristics

• Allow flexible access to the service
• Enable user style sheets
• Optimising colour and contrast
• Using text effectively to improve readability
• Consistent page layout
• Provide content that can easily be scanned
• Easy to use navigation
• Provide meaningful links
• Provide accessible forms
• Use JavaScript carefully
• Promote accessible multimedia
• Provide informative feedback and prompts
• Aid recovery and prevent errors
• Aid Task Completion

3.2.2 End User Evaluation

 

3.2.2.1 Participants

Thirty-two participants took part in this evaluation and were made up from four different user groups.  There were fourteen Blind and partially sighted people, five Deaf and Hard of Hearing, eleven people with learning difficulties and two people with limited dexterity. The participants were also of different nationality, twelve people were from the UK, eleven people were from Portugal and nine were from Cyprus. 

Of the twelve British participants, seven were blind and partially sighted and five were deaf or hard of hearing.  Of the nine Cypriot participants, seven were blind and partially sighted and two were people with limited dexterity.  Finally all eleven of the Portuguese participants were people with learning difficulties.

Of the 32 participants, seven participants were under 21, eight were between 21 and 30, eight were between 31 and 40, six were between 41 and 50, two were between the age of 51 to 60 and one participant was between the ages of 61 to 70.  Nineteen of the participants were male and thirteen were female.

To facilitate a successful end user evaluation, the needs of the users had to be considered and fulfilled.  If the needs of the user were not satisfied, a user or a user group might not have been able to participate fully or at all.  Appendix One highlights the protocol needed to consider the users needs.

Background Information

• All the participants own a mobile phone
• Twenty-nine participants make voice calls and three do not
• Only one of the users did not send and receive texts, however the blind Cypriot users were not able to read the texts as it was an inaccessible function of the phone and some of the Portuguese participants required assistance to read and write text messages
• Only six people used their phone to access the WAP services
• Only fifteen people used their phone to take photographs
• All but four of the participants use their mobile phone to store contacts. 
• Sixteen of the participants use their phone to store events in the calendar and sixteen don’t.

3.2.2.2 Focus Group Phase

 

In order to facilitate an effective group meeting, a plan was written to direct and stimulate discussion among the participants.  This plan consisted of four sections, which consisted of an ice breaker, barriers to using mobile phones, the button design, battery and card, and the final section focused on input/output methods, packaging and instructions, alerts, making calls and finally the services.  Groups consisted of no more than seven people and lasted approximately two and half hours, including breaks.  Once the focus group work was finished each participant was given a mobile phone to use for approximately four days as well as a questionnaire to complete.  The focus group plan can be seen in Appendix Two.

3.2.2.3 Questionnaire Phase

 

The questionnaire was made up of 48 comprehensive questions about the mobile phone.  The questionnaire was split into four sections; hardware, software, services and miscellaneous questions.  The first two sections were further divided into sub sections to analyse various parts of the phone in more detail.  Each participant was given a questionnaire and was asked to complete it whilst they had the phone.  The questionnaire can be seen in Appendix Three.

3.2.2.4 Interview Phase

 

An interview plan was put together which consisted of up to four initial questions, ten tasks and three final questions.  The initial questions asked for general comments about the phone and the final questions asked about the tasks the participant had just performed.  The ten tasks were as follows:

• Inserting a contact,
• Sending a text message (using non-predictive text),
• Sending a text message (using predictive text),
• Taking a picture and sending it,
• Changing the left and right selection keys (Shortcut keys),
• Changing the ringtone,
• Attaching a wired headset or cable,
• Accessing web services,
• Dismantling the phone
• Assembling the phone.

These tasks were chosen on the basis that people need to do on a mobile phone.  The interview was the last stage of the evaluation for each participant and was conducted by an evaluator and where appropriate an interpreter.  The interview plan can be seen in Appendix Four.

 

3.2.3 Comparison stage

The aim of the evaluation was to determine if the heuristics matched the results of the end user evaluation.  In order to do this, a heuristic evaluation was carried out as well as an end user evaluation.  The results from the questionnaires, focus groups and interviews can be seen in Appendix Five.

Following the completion of the heuristic evaluation and end user evaluation the results were cross referenced with each other.  Not all checkpoints could be tested against the results of end user trials due to the amount of time that would have taken.  It was also noticed that not all checkpoints were applicable to the handset that was used for the evaluation.  Therefore it was necessary to catalogue how many checkpoints were applicable to this evaluation before the matching process was performed. Some of the checkpoints would not be applicable as were dependent on the phone having that functionality (e.g. the checkpoints which relate to joysticks could not be evaluated if the phone doesn’t have a joystick).  Table 1.0 in Appendix Five shows the number of checkpoints broken down by each hardware related heuristic principle and table 1.1 shows the number of checkpoints broken down by each software related heuristic principle.

When the applicable checkpoints had been identified the results from the heuristic evaluation could be cross referenced with the results from the end user evaluation.   This showed how many of the applicable checkpoints were tested.  The percentages can be seen in Table 1.0 for the hardware heuristics and table 1.1 for the software heuristics in Appendix Six. 

3.3 Procedure adaptations

 

The procedure was adapted slightly for the participants with learning difficulties as it was recognised that this user group may struggle with completing a questionnaire on their own and may not be able to use a new mobile unassisted.  Therefore for this user group, the questionnaire was amalgamated with the interview scenarios and both were completed at the same time.  As well as this, the participants were presented with a set of pictures representing the rating scale.  This was done because there was concern that the participants might not recognise or have the ability to differentiate from the text based version (“very easy”, “easy”, “neither easy nor difficult”, “difficult” and “very difficult”).

3.4 Project limitations

 

There were a number of limitations that had an effect on the project, of which; the equipment was possibly the most influential factor.  At the start of the project it was hoped that one of the mobile phone operators would supply the project team with up to twenty handsets.  However, this was not possible so RNIB purchased six.  Whilst these handsets were sufficient it meant that the timing of the evaluations and the time users were allowed to use the phone was limited. 

Another limitation was the languages that were available on the evaluation phones.  The phones only had three languages built in, English, French and German.  As some of the participants were not based in the UK this was an issue.  Linked to this factor was the language of the screen reader needed for blind and partially sighted users in Cyprus.  It was not available in Greek for the blind Cypriot participants.  Therefore the Cypriot participants were selected on the basis that they spoke English so they could understand what the screen reader on the mobile was saying.  However it is known that this may have an effect.

The third limitation was the access to participants.  The main user groups that were affected in this instance were the deaf and hard of hearing participants and the people with limited dexterity.  Although a broad recruitment drive was conducted to obtain participants from all user groups, only a very small number of deaf and hard of hearing and people with limited dexterity responded who could actually take part in the timescales allocated. 

The final major limitation was the timescales.  The project had to be completed inside a six month timeframe and this included a trip to Cyprus and Portugal to complete the European part of this evaluation.

Contents

Next chapter