Introduction:
Engineering Education has tremendously benefitted from the use of information technology. One of the key aspects of engineering education is hands-on experimentation, either using the main equipment or by using tools that simulate the equipment. The use of such applications enhances the quality of education as well as accommodates students with various learning styles. (Carlson and Sullivan 1998)
An important aspect of these systems is that it should be easy to use, learn and understand. In essence it should be usable. Usability is defined as “the capability of the software product to be understood, learned, used and attractive to the user, when used under specified conditions.”(Bevan 2001). Usability testing plays an important role in the development of interactive educational software and user-centered design is quintessential for promoting its usage.
In this study a Top-down approach based on a hypothesis (Lynne Cooke) was used, where in we entered the study based on the hypothesis that increase in the time the users looked at help button translated to an increase in learning outcomes. Eye tracking study was used to prove this hypothesis.
Role of Eye Tracking in performing usability evaluations:
Eye tracking is defined as technique in which an individual's eye movements are measured in order to understand where an individual is looking as well the sequence in which he/she shifts from one location to another location. In the usability field these measures would help researchers in ascertaining various factors that would impact the usability of the system. (Alex Poole and Linden J ball)
One of the primary reasons why eye tracking is used in usability studies is because of its relationship with the cognitive process. The “eye-mind” hypothesis (Alex Pool, Linden J ball) indicates that what a person observes is assumed to indicate the thought “on top of the stack” of cognitive process (Just & Carpenter, 1976). This means that eye movement recordings can provide researchers with a trace of the users attention over an interface. Other eye tracking measures such as fixations (stationary eye movements) can indicate the processing time on the area that was fixated (Just & Carpenter). In combination with other usability measures such as speed of task, user behavior (observation,) eye tracking can provide a useful insight into the users cognition. [Goldberg.J.H and A.M Wichansky]
The eye tracking data can be analyzed both quantitatively as well as qualitatively. From a qualitative perspective, eye tracking fixation data can be studies using gaze plots and heat maps. Gaze plots provide static view of the gaze points, which are fixation points that are numbered. These are colored in blue with the radius indicating the length of the fixation. These plots help in visualizing scan paths, which can be defined as “spatial arrangement of a sequence of fixation” (Jacob and Karn). Heat maps or hot spot graphs indicate the user focus using the temperature analogy by using darker colors to represent intense focus and lighter color to represent less focus. This provides researchers with overall attention of the user to various regions of the interface.
To obtain specific quantitative data researchers create areas of interest, which can be defined as “area of display or visual environment that is of interest to the research or design team and thus defined by them (not by the participant)”(Jacob and Karn). These are defined over specific areas of the interface. Researchers quantify the eye movements in this area, with which the visibility, meaningfulness and the placement of the elements can be objectively evaluated. Thus, providing researchers with findings that can be used in evaluating the design of the interface. (Goldberg & Kotval, 1999)
Thus, eye tracking can be used in examining how users adapt to unfamiliar layouts as well as where and how they look for information.
Other Research Techniques used in combination:
Linear Axis Rapid Development Phase-II(fig1a,b,c):
All these method were used in the evaluation of the Linear axis RDS, an engineering software for mechanical students.
The Linear Axis RDS is used in teaching materials on control design/insertion in the Mechanical Engineering curriculum at Missouri S&T. It has a graphical user interface with three main modes: simulate, emulate, and implement. The interface was improved from a previous round of evaluation.
The overall interface was made wider. An interactive help menu was created to provide step-by-step help instructions to the user. This menu was made accessible by clicking a prominently visible help button. A second inclusion was a tip box, which provided the user with simple guidelines on the usage. The third inclusion was feedback box that flashed messages, which guided the user in the initial stages. The latest inclusion was the email button at the bottom through which the student could email their data files.
Research Method:
The participants for this research were senior students enrolled for the “ME 279: Automatic Control of Dynamic Systems” class from section A and B at Missouri University of science and Technology.
Section A was chosen as the experiment group while section B was chosen as the control group. A pre questionnaire containing questions on demographics, learning styles and interest in science was initially collected from both the sections.
The experimental group was asked to develop their controllers and use them with RDS software as a part of an assignment. The control group was asked to email their controllers to a TA.
When a participant arrived in the lab, he or she was shown the setup. The participant was asked to first sign the consent form. The participant was then given a very brief introduction of the RDS and the research techniques. The list of the tasks was then given to the participant. The list was composed of 4 tasks.
TASK 1: Use the Jog function.
TASK 2: Simulate your controller. Generate model.
TASK 3: Emulate your controller. Generate model.
TASK 4: Implement your controller.
The students were given 30 min each to complete all the four tasks.
We used the eye-tracker to record the first and the fourth tasks. The participant was asked to think-aloud when he or she was using the RDS to complete the second and the third task. When the participant completed all the four tasks, he or she was asked to complete a post questionnaire. After the participant completed the questionnaire, an interview was conducted.
A post questionnaire was distributed to the students of section B.
Task Performance:
The below table provides an outline of task performance of the students.
Almost all the participants were able to complete all the tasks without the need for any prompt. Some of the participants who had difficulty with the task had issues with their controller, but none of them had any serious concerns with the software.
Results Summary:
Quantitative Analysis:
Learning Styles:
Eye Tracking Results:
Eye Tracking results suggest the validation of the quantitative data and qualitative data. Fig 4a shows that the user immediately noticed the Jog button as part of the first task in which the user had to use the jogger function. Fig 4b show the user noticed all his selections but skipped the feedback assuming his controller had been loaded. Unfortunately most of the participants assumed that their controller was loaded without checking for feedback.
Analysis of heat map from all the participants suggests that all the areas of the interface received attention. Fig 5 a shows the heat map of the jog function. Fig 5b shows that the help menu received good attention from users.
Figure 6a shows that the user did not read the pop-up message that has important data regarding the saved data file. This was observed in multiple participants, which was a cause of concern.
Recommendations:
Conclusions:
References:
