This report covers the second part of the structured expert review evaluation of a pilot hypermedia application for the Factory Information Resource Management (FIRM) project. The evaluation process used experts from the industrial environment. The evaluation focuses on the usability of the proposed application in the industrial environment. This report presents the rational behind the evaluation and the design criteria to be evaluated. In addition the procedure used and the results of the evaluation are present along with the lessons learnt.

Technical Report No. 98-3 June 1998.
ISBN 0854326723
Copyright © 1998, University of Southampton. All rights reserved.

Contents.

1. Introduction
1.1 Systematic User Evaluation (SUE).
1.2 Defining the evaluation tasks.
1.3 Choice of usability criteria.
1.4 Defining Evaluation Tasks.
2. The Pilot Study Evaluation .
2.1 The Expert Reviewers
2.2 The procedure.
2.3 Comments on procedure.
2.4 Results
2.5 Comments on the Results.
3 Evaluation of the Input Device.
4. Conclusions.
Acknowledgements.
References.
Appendix A Comments By Expert Reviewers

This report covers the second part of the structured expert review evaluation of a pilot hypermedia application for the Factory Information Resource Management (FIRM) project. The first part of the expert review was undertaken to establish the Human Computer Interaction (HCI) aspects on industrial hypermedia system, focusing on the general graphical user interface aspects. The results and process used in the first part of the evaluation are reported in a previous technical report on 'Evaluation of a User Interface Developed for Industrial Applications' [Wills 97a]. The first part of the review was carried out using experienced specialists in the field of hypermedia. While the structures expert review report here was conduct used experts reviewers from the industrial environment and was conducted on site at Pirelli Cables. The lessons learnt from the first review were applied to the review process reported here. The pilot application under evaluation was design and authored using the methodologies described in previous technical reports on Industrial Hypermedia Authoring [Wills 97b] and Industrial Hypermedia Design [Wills 98]. This report presents the rational behind the evaluation, the process used, the results and lessons learnt from the evaluation.

Garzotto et al [Garzotto 97] have proposed a Systematic Evaluation of Hypermedia (SUE) that basically combines inspection with empirical testing. It aims to reduce the disadvantages of the inspection and empirical testing, to produce a reliable cost effective solution. The system can be divided into two phases:

• The Preparatory phase: This is can be a time consuming process in which the conceptual framework for the evaluation is completed. However, this is required to be carried out once only for any particular type of hypermedia application.
• The Evaluation Phase: This is a mixture of ‘expert’ and user evaluations. The results of these evaluations are fed back to the designer for consideration.

SUE was originally designed with Hypermedia Design Methodology (HDM) and museums applications in mind. The details of the evaluation process as given in the literature [Garzotto 97] cannot be directly applied to an industrial environment. There are several reasons for this: -

• The evaluation process is heavily biased towards the Hypermedia Design Model, proposed by Garzotto et al [Garzotto 95].
• The resulting hypermedia application is used in an unrelated environment, which is it is based around a museum environment compared with the industrial environment.
• The system has to respond to a different user base. That is the SUE process was design around application that involved untrained users from the general public.

Hence, the actual abstract task and evaluation methods in the execution phase are not applicable to the industrial Environment. The principles that can be distilled from the preparatory phase of the SUE approach and used to provide a general framework from which a systematic evaluation of a large-scale industrial hypermedia application can be developed are: -

• That the model primitives or design criteria need to be explicitly stated.
• The usability criteria of the application need to be defined.
• The definition of the abstract and concrete tasks needs to be defined.

Even these principles need to be modified before they can be used in the evaluation process. The modifications are necessary to ensure that the evaluation methods used will evaluate the issues involved in large-scale industrial applications and that the language used is appropriate to an industrial environment.

Back to Contents List.

The choice of a model is to guide the analysis of the pilot application and provide a common language/set of definitions for the abstract tasks. In this research a new design methodology is used for the industrial environment [Wills 98]. The main primitives or design concepts that describe the essential features of the methodology are shown in Table 1

. The usability evaluation of a hypermedia application can generally be divided into three main areas.

1. The general look and feel of the user interface. This will include dialogue styles, menus, shape of buttons, usage of colours, window design and the actions the operators have to perform i.e. click on the button with mouse.
2. Evaluation of the representation of the information structure. That is, the Mini-hypermedia Applications (MHAs) structure, change control, access methods, general navigation features including any active media control, for example guided tours, video segments or animations.
3. Evaluation of the application specific information. That is, does it make sense, can the users follow a train of thought, contents of MHA, is information appropriately clustered, are the drawings clear, etc.

The majority of the issues to evaluate in the area of the general look and feel are a function of the propriety software, while information structure and information content are a function of the application domain. Therefore, the latter areas need be specifically evaluated. However intrinsic to the evaluation will be the look and feel of the application as a whole, with special attention being given to those areas of the interface specifically designed for the application, such as is the toolbars and menus.

Back to Contents List.

The usability criteria applicable to the industrial hypermedia environment and to be evaluated are:

1. Retrievability: How easily can a user find the relevant information this is commonly called accessibility, yet this can be confused with the access control in the industrial environment hence the change of name.
2. Orientation: The ability of the user to know their navigational position, the information context and the ability to return to a position if required.
3. Reuse: the design and authoring methodology encourages reuse. However, inappropriate reuse, not reusing enough (or too much) information to make sense in the new context, can cause the user to become disorientated
4. Intuitive. The system will allow the user to grasp the meaning and purpose of the information, dialogue box, window etc. that is being presented.
5. Consistency. Conceptually similar elements are treated in a similar fashion, while conceptually different elements are treated differently.
6. Accessibility. How accessible with/without the correct permissions is the information.

The task definition can be divided into two sections, those for the expert review and those for the general users of the system. The structured expert review tasks (ETs) use the terminology of the model to provide a general description of each task the expert reviews are to try to perform. The defining of the tasks also provided uniformity in the evaluation process, especially when different evaluators are involved. Hence, the evaluation process will depend, more on the procedure used and less on the evaluator's ability. In addition, the task description will identify critical areas and the appropriate usability criteria for evaluation.

The abstract tasks were defined using the following headings:
Tile of Task: - General heading.
Activity Description: - What the user is required to do.
Design Criteria: - Primitives the primitives that are under evaluation.
Usability Criteria: - Those usability criteria that are applicable to the task.
Additional Comments. Here the designer of the evaluation comments on the rational behind evaluating the task.

Back to Contents List.

This structured expert review was conduct at Pirelli Cables. The industrial expert reviews were all from the management team of the Supertension line. In the evaluation the management team was mainly there to review the information structure and the content of the specific information. However general comments on the look and feel of the pilot application were encouraged.

The expert reviewers were all from the management team of the Supertension sheathing line.

ER 1 Manufacturing Manager (Supertension Cables).
ER 2 Maintenance Engineer (Supertension Cables).
ER 3 Principal Process Engineer.
ER 4 Process owner (Supertension Cables).

The expert reviewers were chosen because of their experience in the field of manufacturing cables and not computer science. The age ranged of the experts was between 30 and 55 years old, and they had work in the industry for between 12 and 35 years. All of the reviewers use computers daily and e-mail in the course of their work. In response to questions on their experience of using computer packages such as word processing, spreadsheets and presentation packages:

• Only one of the reviewers classified themselves as an experienced user. This was also the only person to have written macros.
• The other three reviewers classified themselves as either novice or intermediate
• None of the reviewers classified themselves as expert users.

Back to Contents List.

The Expert Reviews were carried out over two session. The first was with ER 4, the second with ER 2 and ER 3. The expert review was given four scenarios recorded as expert tasks (ET) 1 to 4. The experts were asked to make comments whilst carrying out the scenarios, in accordance with the principles set out in section 1.2. The observer took notes and occasionally prompted the reviewer for comments.

The activity descriptions for the scenarios were slightly modified between the first and second group of reviewers. That is the manual that the reviews were asked to comment on were swapped between tasks 2 and 4. This did not effect the design or usability criteria under examination. The reasons for the slight change, being that not all of the recommendation suggested from the first session had been implemented in time for the second review, that is the diagrams were not included. Hence, to encourage comments on the different method use to represent diagrams within the manuals the reviewers were asked to examine a different set of manuals.

Back to Contents List.

The length of time taken to carry out the first review was 46 minutes, this was about the right length of time for an expert reviewer working alone. The length of time taken to complete the second session with two expert reviews was just over 60 minutes. The sessions were initial planned to take 90 minutes based on 3-4 expert reviewers. This time of 90 minutes for 3-4 reviewers was based on the experienced gained while conducting a previous structured expert reviews reported by the authors [Wills 97a]. That is discussion of each scenario naturally generates more discussion and hence would take longer the more reviewers in a session.

The sound recording of the comments was necessary as the reviewer was working along and unless people are used to speaking their thoughts aloud, it can be intimidating to do so, especially as the comments are recorded. The sound recording acts as an aide-mémoire for the observer in writing the report only, therefore all sound recordings are erased once the report is completed. The sound recording of the comments proved useful although there was only one reviewer. The usefulness of the sound recordings was in part due to the observer needed to encourage the reviewer to think aloud. Therefore, the observer found it both difficult to engage the reviewer and write detailed notes on the reviewer's comments especially once the evaluation was near the end of the session. Towards the end of a session the expert reviewer's ideas flowed more easily and hence the notes taken were not detailed enough to obtain the full recommendation of the reviewer. Hence, listening to the sound recordings helped in compiling the evaluation report.

All comments and observations are accurately and factually recorded. However, what is not recorded is the weighting giving to the observation made by the expert reviewer. Hence, time could be spent on correcting some of the less crucial issues, that is the observations that are the easiest to fix or at least the more interesting to the designers. Therefore, a system of classifying the observation is required to ensure that the important issues of usability are identified. Ideally, this should be carried out during the review.

There is always a danger with classification of observations using terms such as, major and minor, or important and not important is that the observations with the lower rating are ignored out of hand. It is however, always a designers/project manager choice as to which observations they will allocate resources to.

A method is required whereby topics the reviewers discuss in passing or between scenarios that are of a general nature and cannot be attributed to any particular scenario, are given full consideration. For example, the comments made about the usability of the touch pad and the drawing issues. These are much bigger issues than the usability of certain scenarios. What at first might appear to be a minor or a not important observation can have wider significance if repeated by different groups of reviewers in different sessions. Therefore, the observer or evaluation team needs to be aware of trends and the significance of comments made by expert reviewers. Hence, it essential the observers must record all comments, especially those that on the surface do not appear to be important. This requirement is in addition to the weighting of observations made by the expert reviewers.

Each of the scenarios was designed to evaluate aspects of the design and usability criteria. As stated earlier the expert reviewers were not computer scientists or usability experts. Hence, the language used to describe to observations could not be mapped directly to the criteria under evaluation. However, the purpose of the review was to assess the suitability of the system for the factory environment and in particular the factory-floor. Hence, their observations made were focused purely from the user's point of view. Therefore, the observation were grouped as below, yet by examining the observation closely and especially the lack of criticism showed that much of the system was transparent to the users.

There were also some positive comments, for example the 'close button' on the toolbar to close all the pages of the manual pages and the thumb nail sketch approach for ensuring large diagrams do not interfere with the reading text.

One of the advantages of carrying out these types of reviews is that they allow the user to get involved in the development process. This was especially relevant in the case of Pirelli Cables, where the original senior manager who endorsed the project were promoted and moved to a different site. Hence the review proved to be a successful mechanism in which the line management could be involved in the development and begin to become owners of the project concepts.

Back to Contents List.

The observations made by the reviewer were grouped into 3 categories, those referring to: -

• Content: - these observations were to do with the actual content of the information. That is missing links or nodes, etc.
• Presentation:- these observation were to do with presentation and navigation of the information
• System: - these observations were to do how the user interacted with the application software.

In the categorisation of the observation, some of the observation can clearly be place in a number of categories. Hence, the observations are placed in the different category that best represents the main point of the observation. A full list of the comments is included in Appendix A. However, the observations that most concerned the expert review can be summarised as follows:

• There were a number of occurrences where the system showed links to a number of bitmaps and documents, yet the reviewer was unable to follow them. While the error message was explicit 'no document found', the system still opened a blank viewer, which could not be closed until the reviewer exited the system.
• The other observations were requests to add additional links.

Presentation.

• Displaying of a bitmap from the thumbnail sketch was cumbersome, that is, the bitmap was wider and deeper than the screen. This will confuse the operator, as it is difficult to resize the window, due to it being so large.
• There were a couple of observations regarding the use of the bitmap viewer. The first was concerned with the initial start-up position, this can be fixed by setting the default position. The other was to do with a losing the tool bar when resizing an application windows.
• The table and figures that are embedded into the document were not completely visible, that is only part of the table or figure could be seen as no horizontal scroll bars where available on the viewer. This was exacerbated by the fact that the screen layout was under SHEP control. Hence the Shepherd would not allow the window to be resized.
• How with a 17 inch monitor were we going to display large drawings (A0) containing lots of detail, to complicate things further what will happen if the maintainers needs a to view a manual at the same time.

The other observations were to do with the presentation of the material using SHEP or the size of the documentation. For example how to control the page turn so that drawings that are represent by thumbnails are included.

System.

• The main concerns were with the logging process. The reviewers felt it should be as simple as possible and without the user needing to know how Microsoft Windows operates. For example one request was could we set the system so that the reviewer logs in once (into Windows) and the system logs the operator into FIRM and auto starts the application.
• A few of the observations referred to functionality in terms of buttons to the toolbar. For example to clear the screen and return to start, to jump back to a document the user marked as interesting, etc.
• The information is required at the point of use.
• The Glide pad as an input device was too sensitive and hence awkward to use.

To correct some of these observations require various tools, and this review has proved help in identify some of the tools needed. There are several of the critical observations that could not be rectified, as they would involve change the source code to Microcosm or Window's NT. For example when the reviewer zoomed into a diagram using the bitmap viewer, the window would resize itself. In doing so, the task bar was not fully displayed. Only the first few icons could be seen and used, i.e. tool bar does not wrap. This is a function of the bitmap viewer and is comment to many Windows' applications (i.e. Microsoft office).

The problem where the tables and figures that are embedded into the text document and could not be read as the viewer is made smaller due to no horizontal scrollbars is a 'bug' or design flaw of the RTF viewer. Multicosm are due to release a new multipurpose viewer in 1998, this will be able to display over 50 different file formats. The author of this thesis has evaluated the viewer during its beta phase, the viewer did manage to display text and RTF formats adequately, but at that stage did have problems with Word documents. In addition it did not have the same functionality as the current DXF viewer for vector based drawings. While it solves the problem of embedded figures and tables, it looses the advantages of the present RTF hypertext viewer, that is, it follows the paper paradigm, with page breaks and headers/footers. Hence more time is spent ensuring that the layout of embedded figures, diagrams and paragraphs do not stretching across page breaks rather than in the most appropriate place to give them meaning.

The guided tours in the pilot application were used to maintain the original structure of the manuals and procedures. Guided tours were not effectively supported in the administrative layer. However a method was developed where by the information from the guided tours was used within the FIRM authoring environment to enable the user to step through the original structure as thought they were pages in a manual. This method is better than the guided tour as it allow the user to step back as well as forward and the user can jump into any part of the tour. Predefined tours are part of the wider issue of how the user navigates a large industrial application and associates nodes of information, again braking the confines of the paper paradigm.

The observation that ideally the system needs to be taken to the point of use, is a very valid point and is mainly an issue of the choice of technology. The issues raised about the point of use and the ability to view all the information has far wider implications and requires further work. Although a maintainer may have opened in front of them a number of diagrams and books, they will only be reading one at a time. However, they are able to glance from one to the other. This is essentially a combination of the issues of navigation and HCI for technical information. Hence, there needs to be some form of virtual 'bench top' in which all the information can be laid out. This problem will be partly solved by very near future technology. For example the release of operating system like Microsoft Office 98™ with it ability to display information on two screens. In addition TFT (Thin Film Transistor) and LCD screens are becoming cheaper. These have the advantage of being slim line (1 cm thick) very light compared to the weight of a CRT screen with the same viewing area of a CRT without the need for the 40 mm surround that is required to hold the CRT in place. Hence it is very feasible to have two TFT screens fed from a single lightweight PC. Research into how this can be practically achieved and the navigational paradigm to achieve it is required.

Back to Contents List.

The ability to use the touch pad effectively could just have been a matter of practice. However, all three expert reviewers found it cumbersome, and were concerned about how the factory floor users would cope with using the touch pad. Pirelli insist that computers on the factory-floor be housed in a purpose built enclosure, hence the touch sensitive pad (called a Glide pad) was initially chosen for the pilot application, as space on the tray holding the keyboard was limited. In addition a conventional desktop mouse would soon become unusable on the factory-floor due to the ingress of dirt, grease bitumen etc. Hence, the different options were evaluated. The options immediately available were:

• A conventional mouse
• The Glide pad.
• An Electronic Pen and Pad
• TrackMan Marble FX supplied by Logitech
• Touchscreen add-on manufactured by Carroll Touch.

The conventional mouse was used as a base-line device, as all the reviewers had experience of using the device. Each of the reviews attending the session (ER1, ER3, and ER4) tried to use the devices to move the cursor, follow links and select text. The reviewers were then asked to fill in a questionnaire consisting of the following questions for each device. A score was then given to each of the answers as shown in Table 3.

The questions were asked in such a way that the reviewer had to make a choice. In addition the question were structured so that not all the favourable choice were in either the extreme right or left box. The results of the scores given by the reviewers are shown in Table 4.

The reviewers were also asked to rate the devices in accordance to their preference for use on the factory-floor. The first choice of all the reviewers was the TrackMan. To use a TrackMan and keyboard requires an extension of the width of the tray that holds the keyboard. A previous visit to the factory-floor revealed that it was common practice to leave the keyboard tray extended. Hence any modification to the width of the keyboard tray would not effect the common work practice, although it would stop the tray from being places back into the enclosure.

Some of the other general points raised by the reviewers were:

• The pen and pad was felt to be "20 times more frustrating" than the glide pad.
• The touch screen lost out, due to having similar problems with the glide pad, in that if the user accidentally touch the screen with any other part of the hand a false reading took place. That is the system highlighted unwanted piece of text or jumped to a different location other than where the user was trying to move the cursor. In addition, the touch screen suffered generally from parallax error and out of alignment near the edges of the screen. These problems may have been compounded by the use of an add-on touch screen rather than a built in touch screen. One of the reviewers also expressed concern with the users getting tiredness in the arms from using the touch screen, this is a known problem with touch screens [Preece 94].

Back to Contents List.

The evaluation of the Eastleigh application used experts from the industrial environment rather than HCI specialist. Hence the observation made were more on the general feel and the content of the information. It was found that the analysis of the results must include a weighting categorisation of the observation and not just a factual recording. In addition the observer of the reviews had to pay close attention to general comments, that could not be assigned to a specific scenario and comments although seemingly insignificant, but when repeated several times by the reviewer increase their importance. An example of this was the comments on the unsuitability of the glided-pad as an input device for the factory floor. Therefore an evaluation of a number of input devices was undertaken. In the opinion of the reviewers the TrackMan was the best solution. In addition the observations of taking the information to the point of use and how best to view large detailed drawings and manual at the same time, require further work.

Back to Contents List.

The authors acknowledge the EPSRC (Engineering and Physical Science Research Council) for funding the work under grant number GR/L/10482.

Back to Contents List.

The observation made by the reviewer fall into 3 categories, those referring to:-

Copyright © 1998, University of Southampton. All rights reserved.
Back to Contents List.