Evaluation of AIMS-Academic Information Management System.

Wills GB, Hughes GV, Hall W.

Multimedia Research Group Technical report No M99-5

ISBN-085432700-2

Copyright © November 1999, University of Southampton. All rights reserved.

Contents

Abstract

1. Introduction.

2. Methodology for the hypermedia application (expand)

3. Results from the AIMS User Evaluation. (expand)

4. Discussion.

5. Conclusions

Acknowledgement

References

Appendix A AIMS Questionnaire

Abstract

The evaluation process reported here used academic secretarial staff from the Electronics and Computer Science (ECS) department to evaluate the Academic Information Management System (AIMS). The evaluation focused on the subjective opinion of the users and measured the time taken to perform the major functions of the system. This report presents the methodology and describes the rationale behind the approach used. The results and conclusions of the evaluation are reported.

1. Introduction.

The successful introduction of any new equipment or system is largely governed by the acceptance by the users. There is therefore a need to evaluate the effectiveness of the Human-Computer-Interaction (HCI) within the office environment, along with the design of the user interface and navigational aids. For the purpose of the research a user is defined as any person, who would normally have access to the information in the course of their daily work.

Evaluation is concerned with the gathering of information about the usability (or potential usability) of a system, in order to improve features within an interface and its supporting material or to assess a complete interface [Preece 93]. Due to the availability of resources and costs limitations, traditional experimental and usability testing involving a specialist usability laboratory are not an option for evaluating interface issues in this research.

The methodology used to evaluate AIMS evaluating is presented, along with the rationale for and a brief description of the methods used. The report presents the results and conclusions from the evaluation.

Back to Contents List.

2. Methodology for the hypermedia application.

The methodology used for evaluating AIMS is principally based on the methodology used in Factory Information Resource Management (FIRM) project [Wills 99] which in turn used the SUE (Systematic Evaluation of Hypermedia) framework [Garzotto 97] for the user task evaluation. SUE was originally designed with Hypermedia Design Methodology (HDM) and museum applications in mind. Hence, the actual abstract task and evaluation methods in the execution phase are not applicable this toresearch. The principles that can be distilled from the SUE approach, and used in this research, are that the model primitives or design criteria require explicitly stating, and the usability criteria and user tasks defined. The evaluation of AIMS takes place in three stages:

1. Contextual review of the current office practice into which AIMS is to be used.
2. A structured expert reviews is to be carried out using discounted usability engineering.
3. The users perform predefined tasks using AIMS to evaluate the effectiveness of the application. After completing the tasks, the operators fill out the questions the results of which. .

2.1 Choice of users.

It is essential that the evaluation of the hypermedia application takes place within the environment that it is to be used, using the appropriate end users [Yamada 95]. In addition, the choice of subjects is vital to success of any experiment [Dix 98]. Hence, when carrying out the evaluation, different personnel are required for each stage. This is especially applicable to the expert review of AIMS, where both the content and user interfaces are to be evaluated.

When evaluating the user interface experts with computer science and HCI backgrounds should be used. When evaluating the content of the application it is advisable to use experts that are or have worked in an academic environment, preferable connected with administration. However, as each department has its own culture, actual work practices and subtle changes in terminology, ideally personnel should come from the actual working environment when evaluating the content of the application.

The choice of users for the time trial will be the actual personnel (or subgroup) from the academic secretaries, from within ECS that contribute to the information held on AIMS .

2.2 Contextual Review.

Any developed hypermedia application should be relevant and specific to the application domain in which it is to be used. Therefore, in order to produce AIMS it was essential that the current work practices and documentation structures, together with how they effect the content of the application, were understood. To achieve this, an audit of the information structure and working practices, using the contextual inquiry method [Preece 94] was undertaken. Contextual inquiry is an evaluation method that comes under the general heading of interpretative evaluation. It is used in the collecting and analysing of data on how people use technology in their natural situations. The rationale behind this type of evaluation method is that the working environment and practical tasks required of the user will effect their behaviour. Contextual enquiry is used to aid the developers of the hypermedia application in understanding the working environment, to uncover hidden work practices, and to spotting implicit work practices, etc.

2.3 Structured Expert Reviews.

The resources available for the evaluation lends itself to Structured Expert Review [Preece 94], in particular discounted usability evaluation. Discounted usability evaluation is a heuristic evaluation and was first proposed by Nielsen [Nielsen 89] and subsequently refined [Molich 90, Nielsen 92, Nielsen 94]. Discounted usability evaluation can be carried out with few resources, consisting of scenarios, simplified think aloud and heuristic evaluation. The whole cycle needs only a few testers as any more do not produce any significant benefit. The method uses small scenarios which can be easily changed, and as the ‘think aloud’ is carried out informally it does not require psychologists to analyse the results. The advantages of this method are that the majority of errors can be found without having to waste users time. However, the method does rely on the ability of 'experts' to judge the reactions of the user and hence not all the problems will be found. Only a few guidelines are required for the heuristic evaluations;

• Use simple and natural dialogue.
• Speak the user’s language.
• Minimise user memory load.
• Be consistent.
• Provide feedback.
• Provide clearly marked exits.
• Provide shortcuts.
• Provide good error messages.
• Prevent errors.

However, Wills et al [Wills 97] noted that an area that is particularly applicable to hypermedia and not covered above is that of navigation through the information space. Therefore, another principle needs to be added, that of 'Provide Navigational Aids'.

Back to Contents List.

2.4 Defining the evaluation tasks.

The Aims model is used to guide the analysis of the application and provide a common language/ set of definitions for the abstract tasks. The main primitives or design concepts that describe the essential features of AIMS are shown in Table 2-1. The usability evaluation of a hypermedia application can generally be divided into three main areas.

1. The general look and feel of the user interface. This will include dialogue styles, menus, shape of buttons, usage of colours, window design and the actions the operator has to perform i.e. click on the button with mouse.
2. Evaluation of the representation of the information structure. That is, revision control, access methods, general navigation features.
3. Evaluation of the application specific information. That is, does it make sense, can the users find the information they need, etc.

Table 1 The Design model primitives.

The majority of the evaluation issues are concerned with information structure and information content. However intrinsic to the evaluation will be the look and feel of the application as a whole, with special attention being given to those areas of the interface specifically designed for the application, such as the toolbars, menus and data entry forms. The usability criteria used in this research are similar to those suggested by Garzotto [Garzotto 97], there are:

• Accessibility: How easily can a user retrieve the relevant information with/without the correct permissions in the information.
• Orientation: The ability of the user to know their navigational position, the information context and the ability to return to a position if required.
• Reuse: the design of AIMS encourages reuse. However, inappropriate reuse, not reusing enough (or too much) information to make sense in the new context, can cause the user to become disorientated
• Intuitive. The system will allow the user to grasp the meaning and purpose of the information, dialogue box, window etc, which is being presented.
• Consistency. Conceptually similar elements are treated in a similar fashion, while conceptually different elements are treated differently.

  Back to Contents List.

  ---

2.5 Evaluation Tasks.

The object of the time trials was to measure the effectiveness and ease of use of AIMS in entering documents and locating information. The tasks were chosen to reflect this, by using a set of prepared documents.

The defining of the tasks also provides uniformity in the evaluation process, especially when different evaluators are involved. Hence, the evaluation process will depend, more on the procedure used and less on the evaluator's ability. In addition, the task description will identify critical areas to be evaluated and the appropriate usability criteria for evaluation. A list of tasks is given in Table 2-2.

Table -2 A list of the Tasks for expert reviews and timed trials.

Back to Contents List.

2.6 Interviews and questionnaires.

In this research, both flexible interviews and questionnaires were used to gather and analyse qualitative data, that is, the users’ opinions. The questionnaires were designed to measure the scale of: Impression, Command, Effectiveness, Learnability, Helpfulness and Navigability. See Table 2-3 for definitions of terms. The first five scales are based on the definitions used in the Software Usability Measurement Inventory (SUMI) [Hirst 95]. The last one, navigation, was a scaled added to measure an area specific to hypermedia.

Table -3 Questionnaire scales and definitions.

The questions used to measure the criteria in Table 2-3 can be mapped onto the Technology Acceptance Model (TAM) [Davis 93, Davis 96]. TAM is used to measure the user’s:

• Intended use of the technology.
• Perceived ease of use.
• Perceived usefulness.

Two questions used to measure the impression criteria were mapped onto the intention to use, all the questions used to measure the effectiveness criteria were mapped onto the perceived usefulness, and perceived ease of use is mapped from all the questions in command and learnability.

The questions were designed by the authors to reflect the specific application domain. The questions on the questionnaire were arranged so that answering the questions by putting a cross down one of the columns would produce a score of zero. The users were forced to make a choice as there was no neutral answer. As the users were forced to make a choice with each question, the number of questions was limited, see appendix A for full questionnaire.

Back to Contents List.

3. Results from the AIMS User Evaluation.

3.1 Contextual Review.

During the evaluation, many if not all of the secretarial PCs were swapping, indicating that these machines do not have enough memory and in some cases not enough disk space. This may be the result of upgrading to Windows NT 4 and Office 97.

Sharing information.

One of the secretaries does not type minutes, her role is to enter the minutes into AIMS, from a floppy disc. This high lights some of the problems in sharing information in ECS. While there are workgroups set-up for each group, the secretaries are not trained in how to share information through work groups. Therefore, some of the secretaries share information by e-mailing each other, or by copying files onto floppies and passing them to each other. In addition, as there is no enforced departmental standardisation of word processing packages, sharing of information can be difficult.

Achieving information.

The main method of archiving is in the paper format. Some secretarial staff keep their own electronic archive normally, on their hard drive, only lasting a couple of years (approximately 6 sets of minutes) the other files are delete or used as a template for new ones.

Where as in the Electrical Engineering Department, each of the secretarial machines have had a initial predefined directory structure so that administrative information can be easily located and archived.

Distributing of Minutes.

The minutes and any accompanying material is photocopied and mailed to all members of the committee. However, academic staff frequently ask for another copy just before the next meeting. On the surface this seems a straightforward process, however, this is not always the case, for example in the case of the Board Minutes or Academic Committee Minutes, where there can be a considerable amount of additional material. This additional material is often only available to the secretaries in paper format and photocopying and collating the information can be very time consuming. The length of time required to photocopy, collate and put the documents into envelopes, stick the address labels on (which have previously been printed) then walk them to each group, takes between six and eight hours.

Use of the Campus network.

The secretaries mainly use the network for e-mail and finding peoples e-mail address and for general web browsing (outside working hours).

Back to Contents List.

---

3.2 Expert Review

The reviewers were Dr L Carr and Dr K Martinez from the Multimedia Research Group, at the University of Southampton. The expert reviewers have considerable experience of HCI and user interface design. The following subsections presents the comments from the expert users.

3.2.1 Reviewers Comments Scenario: Task 1 Enter a new document

• On the first attempt the process could not be completed with the proxy turned on. As the proxy is only required for scenario 5 it was turned off.
• "How do we log-in?" was the first comment as no instructions were available as to how to log in or submit a new document.
• During the search for the correct page, the reviewer visited the help pages, where no information was available. These need to be in place before untrained users start to use AIMS, i.e. non-secretarial staff.
• After trying a number of options, the reviewers eventually went into "My Document", which requires a password. From here they were able to submit a new document. Hence a clear link to log-in is required. As logging-in is a top-level action they suggested that the command should be on the sidebar.
• Having put a button for log-in, they suggest that underneath the button should go the 'view my-document' and 'submit new document' buttons.
• Following the link from "My Documents" on the sidebar, a dialogue box comes up that requests a password. If the cursor is on the dialogue box and the user enters an incorrect password and the user presses return it goes to "no page found". However, in the same scenario but this time the user clicks the return button, the dialogue box come back with no message.
• Having entered the document details and the file locations, the experts click on the "Submit Document" button. The screen went blank, no other feedback given so they went back and resubmitted the document. Hence two identical documents are in the system. Could some feedback to the user be given while the original document is being converted to HTML.
• The system takes a finite time for the document to be imported during which no feedback is given to the users, and as a consequence the reviewers submitted the document twice.

3.2.2 Reviewers Comments Scenario: Task 2 Update a document.

• The reviewers proceeded to the correct form by following the link 'new version' from the menu in My-Documents.
• Which date do you amend to show that the document is a revised version of an existing document? What tells the users that the document is an amended version of an existing document when viewed in my-documents menu or all-version menu.
• On the sidebar are instructions for a New Edition and a New Revision. Neither of these terms are used on the menu to update a document.
• Should the instructions on what new edition and new revision mean be on a navigation sidebar? Could they be placed elsewhere for better effect?

3.2.3 Reviewers Comments Scenario: Task 3 Search for a document.

• Following the search button link on the sidebar accessed the form used to enter the search terms. There were no problems with this or carrying out the search.
• The search button is sandwiched between to other buttons, Current Versions and All Versions. The other buttons are in fact tables of contents, this is not clear from their name. A solution is to separate these buttons and change their names.
• In addition what is meant by current version.
• The reviewers felt that the width of each column of information may hinder a PDA user from effectively viewing the information, even while in minimalist view.
• When viewing a set of minutes under all-versions, the menu expand in such a way that the sidebar scrolled upward leaving no navigation tools for the user, unless they scrolled back to the top of the page.
• The board minutes are not in date order. Some of these are 12 screen lengths long, can these be slit into smaller nodes?
• Some documents are in PDF format only. However there is link to what should be a Full HTML text document, instead the user is presented with a page referencing the PDF version. If consistency is to be maintained the system needs to clearly identify documents that are only in original or PDF formats.
• Why are ECS and AIMS home on the sidebar? Do they change with the user, would the button be more appropriate under support rather than by themselves.

3.2.4 Reviewers Comments Scenario: Task 4 Edit document details.

• Arrived at the correct form by following the link edit-details from My-documents. No problems with updating the information.
• When you visit a document the option to edit the document, although displayed, is unavailable. The option to edit a document only needs to be displayed, if and when a person with the correct permissions is logged on the other system.
• On the My-document page, the Next/Previous/Expand/Contract options do not always apply, as there are no documents displayed, could these options be displayed only when required?
• The arrow is not a common metaphor for Windows users. This may confuse novice users.

3.2.5 Expert Scenario: Task 5. Following links with WebCosm.

• Required the proxy to be reconnected via a setting on in the option for the browser. This is not really convenient for a general user. If this is a useful feature it should be build into AIMS.
• Links only appeared on names that had on initial before the surname. In addition, where there was a space between first and second initial, the proxy linked to the incorrect personal information.
• No instruction or method of logging off. This is especially important if workstation ia used by other, those authored to change document do not want to leave it open for other to enter data.

3.2.6 General comments from reviewers.

• Generally quick but it will require faster feed back on import.
• It is much easier to use that is view documents than to added documents.
• Terminology inconsistent (version/revision/edition)
• Placing awkward (left hand side bar)
• Help pages unwritten.
• All these remarks are tempered with the "null option" what would I do without it?
• I cannot find this information UNLESS it is on AIMS, that is my firm belief. Not because I am a fan of the software but because I have experience of the department.

  Back to Contents List.

  ---

3.3 User Trials

3.3.1 General comments raised by the users.

Comments raised by the users during the evaluation were:

• The users would like a way of logging out, as closing the browser and reopening it again shows that the person is still logged in.
• All users thought that the ‘submit new document button’ meant submit next/latest edition of the document. Where as ‘submit new document button’ actually means start a new category. Hence could this be called start a new folder, or a similar term.
• Some uses felt that the button on the side bar started too high up, they preferred that they started below the title bar.
• While the functionality offered by Webcosm is of benefit to the users, the procedure required to initiate the functions was complicated and beyond most of the secretaries, could this be simplified.
• Some people were unsure what they would use the system for, this was reflected in the questionnaire scores.

3.3.2 User Questionnaires

A list of the question with number of users responding agreeing or disagreeing and the number who did not answer is shown in Table 3-1.

Comments from questionnaire.

Very few addition comments were made. Those that were made, came from those that had been at the university for approximately one year.

Impression- user's feelings or emotions when using the software.

• I just have to get used to using it.
• I do believe that staff using this system would benefit from using this system, they would also benefit from training and sharing experience with other "new" users.
• I do not agree that the system is awkward to use, once used regularly I feel it would be a useful tool.

Command - the measure to which the user feels that they are in control:

• I felt in control, but commands were slow taking place, it could be my machine though.
• I did not feel completely in control, not a lot of use on system has left me feeling not too confident.

Navigability - the degree to which the user can move around the application.

• I would need to use the system often to familiarise myself with this application, I could probably answer the questions more fully after using the system regularly.

Learnability - the degree to which the user feels that the application is easy to become familiar with.

• More time and daily use would help me to feel confident with the application.
• The evaluator was helpful, but due to constant interruptions, phones, etc, difficult to concentrate in office environment.

Helpfulness - the degree to which the application assists the user to resolve a situation

• Most of the users were unable to comment on this aspect of the system as they did not encounter any problems and did not refer to the help information.

Effectiveness - the degree to which the user feels that they can complete the task while using the system.

• Software was effective. Would help me do my job effectively.

Other comments.

• Time and daily use would help me become proficient on the system
• From a personal point of view , would be for me to access the system regularly, even if only just to peruse.

Shaded area indicates a negative response

Table -1Question and response of users, by catergory.

Back to Contents List.

Sores from the Questionnaires.

The questions were grouped under the appropriate criteria, the score are shown in Table 3-2, the table also shows the normalised score. The scores were normalised by dividing the score by the number of respondents and the number of question they answered. The maximum normalise score is +1 that is the users very strongly agreed to –1 they very strongly disagreed to all positively phrased question (and visa-versa on all the negatively phrased questions). A sore of 0.5 indicates that all the users agree with a positive statement, i.e. They agreed with ‘The AIMS system is one that I want to use on a regular basis.’

Table -2 Score from the questionnaire.

The TAM criteria scores are shown in Table 3-3.

Table -3 Technology Acceptance Model Scores.

The questionnaire also showed that the scores given was not effected by the user access to the internet from home, or the users preference for using the internet. However, the over all score from those users that had been at the university for approximately a year were lower than those who had been working at the university for a longer period, see Table 3-4. In addition all these people were over 35 years old, however other personnel in this age bracket rated the system highly.

Table -4 The scores according to length of service

The difference in length of service also effected the perceived usefulness and ease of use, see Table 3-5, the normalised score are calculated in the same way as for Table 3-2.

Table -5 The effect of length of service on user perception (normalised scores).

Back to Contents List.

3.3.3 Time taken to complete tasks.

The average time to complete the tasks is shown in Table 3-6.

Table -6 Average times to complete the tasks.

Back to Contents List.

4.0 Discussion.

The general comments from the users showed that they were interested in the idea of a central repository for administrative information. This was reflected by the high score from the questionnaires given to the Impression Category, Table 3-2, and the intention to use category from TAM criteria, Table 3-3. All the users replied that they would recommend the system to their colleagues, and seven of the eighth users said that they would want to use the system regularly.

However, nearly all could not envision its use beyond just the current system usage. In addition, there appears to be little encouragement from the management or academic staff to use the system beyond it current usage. This will need implementing from above and an active program by senior staff. In addition, the staff felt that they could do with more practice to use the system.

The amount of memory in the PC’s hinders the progress of many tasks, coupled with and the lack of training in IT especially in the area of sharing information. This impedes the efficiency of the secretarial staff.

The user generally thought that the AIMS system would be of use in their work, the Effectiveness/Perceived Usefulness criteria score, shown in Table 3-2 and Table 3-3, reflects this. Similarly, from these tables, the scores show that the majority of the uses found the system easy to use, represented by the command and learnability category. The low score in the Helpfulness category was due to all the users not experiencing problems during the evaluation, see Table 3-1. Hence some of them felt that they were unable to comment on his section.

The results from the questionnaires showed that the majority of users found that they could navigate the information space easily see Table 3-1 and Table 3-2

The users perception of the system varied with the length of service within the university see Table 3-4 and Table 3-5. Although new to the university the users were experienced secretarial staff. However, their perception, especially on the usefulness of the system, differed significantly from staff that had been at the university for some time.

The time taken to complete the tasks were generally consistent, throughout the user group. The obvious saving of time comes from the ability to publish the set of minutes extremely quickly, when compared to the current method. The ability to disseminate information quickly would be of benefit to the department. However, this will only be an advantage if there is a paradigm shift. That is that the academic member of staff will need to ready the minutes on line, or as last resorts print them off to read them.

Back to Contents List.

5.0 Conclusions

As a method of archiving and disseminating of academic administrative information AIMS is superior to the method presently used. That is, it is faster to disseminate information and a wider set of documentation is more easily accessible to all secretarial staff.

Generally the users perception is that the system would be useful to them in performing their job function, and they intend to use the system. However, this perception was less in those who had worked in the university a relatively short period of time.

There is a requirement for a program of expansion for the current document set used in AIMS. In addition, a review of the computer administrative support/policy for secretarial staff is required, if they are to continue to operate and share information efficiently.

Acknowledgement

The authors acknowledge Joint Information System Committee (JISC) for funding the work under project number 462.

Back to Contents List.

References

Back to Contents List.

Appendix A AIMS Questionnaire

Master Questionnaire, used to aid data entry

Name: ...........................................................e-mail name......................

User status: Missing Value 9

Age:

Approximate Length of service within University: ..............Years...........Months

e.g. 0 years 4 months

Experience of using the Internet/World Wide Web:

Do you use the Web at home:

Do you use the Web at work (not just for work related information):

Do you use ftp files to transfer files?

Back to Contents List.

Any other addition comments you wish to make?

Back to Contents List.