Interaction logging

Interaction, or software logging involves the recording of key presses, mouse movements and touchscreeen icon selection to determine how users interact with a system when conducting different tasks. Data is collected and analysed using specialist software, which is synchronised with video and audio logs to provide a picture of the users' actions during task completion.

The advantage of this type of evaluation is that it is unobtrusive and provides a large amount of data on users' actions for comparatively little involvement on the part of the evaluator. The large amounts of data derived from interaction logging, can, however, be immensely complex and time consuming to analyse. This is becoming less of an issue as more sophisticated analysis software is developed, however, this in turn can lead to a situation where considerably more data is collected than is required, simply because it is easy to do so.

Interaction logging also presents ethical issues. Sophisticated software packages mean that it is becoming increasingly easy to gather information on increasingly subtle and diverse aspect of users' actions. The problem arises when the data is unrelated to the evaluation, or even more importantly, when personal information is obtained. Informing people that their actions are being logged is one way of addressing certain ethical issues, but this might affect the way a user interact s with a system. Another technique is to only collect information that is relevant to the evaluation, thus avoiding the temptation of collecting and analysing large swaths of data until a relationship is found.

When can interaction logging be used?

Interaction logging can be used at various stages of the development process. The technique is often used to evaluate software-system prototypes prior to full implementation. Interaction logging is also commonly used evaluate high-fidelity prototypes and during summative evaluation following implementation of the final product.

How to conduct an evaluation involving interaction logging

The type and level of sophistication of the logging software will dictate what information can be gathered; however, ultimately the software is simply a tool for collecting information. As such, the general principles involved in this type of evaluation are the same as for any user testing. These are:

Identify the users and the sample tasks

Early in the development process it is advisable to focus the evaluation on how users complete specific tasks, pre-defined by the evaluation team. This limits the amount of non-critical information gathered, which can mask the findings of the evaluation. As the system or product is further developed, specific tasks remain of interest, but focus should now shift towards collecting data on how users 'use' the system as a whole.

When conducting the evaluation it is important that the users are representative of a broad cross-section of the community. This increases the percentage of usability problems that the evaluations will identify.

Ensure the correct recording equipment is available and functioning

Although it may seem obvious, it is important to ensure that the logging software is comparable with the system under evaluation. Information available from one software logging package may not be available from another. Furthermore, if a product is bespoke or early in development, it may be that specific add-ons to the logging software are required, to ensure that the relevant data are available from the evaluation.

Identify the project aims (and stick to them)

As with any evaluation, the temptation to collect data not relevant to the project, simply because you can, should be avoided. Current logging software make it possible to collect and analyse large amounts of data with ease. If, however, the context in which the data was collected is not known, it is very easy to generate potentially erroneous findings.

It is also important to ensure that, simply because software logging can be performed unobtrusively, user consent is not ignored. If information about individual users is collected, consent to use this information should be sought.

Conduct the evaluation

The evaluation should be conducted in a similar fashion to any observational analysis. When the subjects first arrive, they should be briefed on the nature of the study and purpose of the evaluation. Depending on the type of evaluation (i.e. rigid task-based or open-ended), subjects should be thoroughly briefed on the tasks to be completed and consideration should be given to providing an opportunity for the subjects to familiarise themselves with the system under evaluation. This is particularly important since the evaluator is unlikely to be present during the evaluation.

Although the evaluator need not be present for the evaluation, it is good practice to be on-hand should queries arise, particularly if the evaluation is more prescriptive.

Collect & analyse the data

Interaction and software logging systems automatically collect the data; add to this a series of tools now available to analyse and explore the data both quantitatively and qualitatively and the collection and analysis of the data may seem straightforward. It is important, though, to ensure that the audio and video logs are synchronised with the data from the software logs, so that a complete picture of the evaluation can be built up. This, in itself can be a time consuming process. In addition, whilst powerful tools to explore and analyse the data exist, the data that is produced is often highly complex and significant effort may still be required to distill the information for an evaluation report. AS such, it is important that sufficient time is allowed for analysis and reporting of the data.

Debrief

As with any evaluation, once the data collection and reporting is complete, you should return to the research objectives and critically review whether the evaluation met the objectives. The findings from the study can then be used to refine the product or system where necessary. The decision may then be make to conduct further evaluations, as the prototype is developed further. Finally, the outcomes from the study should be shared (where relevant) with the the users.

How many evaluators are required for interaction logging

Since logging of the user actions is performed by the software package, it is normally only necessary for one evaluator to be present (or available) during the evaluation. It is, however, a advisable to develop the tasks in conjunction with other experts, to ensure that the task structure and content is varied and captures all those aspects under review.

Advantages and disadvantages of interaction logging

Advantages

Data collection is unobtrusive;
It is possible to gather large amounts of data for comparatively little evaluator involvement;
Data are automatically logged and time stamped, meaning data is simple to find;
Modern tools make it possible to analyse large amounts of data relatively quickly and easily and present the data in an increasingly user-friendly way.

Disadvantages

Temptation to collect every piece of available data (even if not relevant), leading to an unfocused evaluation;
Numerous ethical considerations associated with the ability to collect data without the users knowledge;
Logging software and programs can be expensive and still require modification to be fit for purpose for the specific evaluation.

Return to Observational Analysis page

References

The information contained on this page was taken from the following sources:

Preece, J., Rogers, Y. & Sharp, H. (2002) Interaction design: beyond human-computer interaction. New York: John Wiley & Sons.
Okada, H., Ashio, T., Kunieda, K. & Shimazu, H. (2004) Interaction logging and analysis tool estimating expected operations for unexpected user operations. In Proceedings of the Human Interface Symposium, 2004. Japan, 2232.