Abstract
This paper describes the RITE (Rapid Iterative Testing and Evaluation) method used in usability testing of software products. It covers the benefits and pitfalls of using RITE methodology along with comparison with other usability methods.
Table of contents
Abstract 1
Introduction 3
Case study: First use of RITE method 4
Case study: SMS 2.0 - Converting SMS language into Formal language 5
Benefits and critics of the RITE method 6
Introduction
The RITE (Rapid Iterative Testing and Evaluation) method was introduced in 2002 by a team of Microsoft professionals working for MS Games Studios. This method defined by Michael Medlock, Dennis Wixon, Bill Fulton, Mark Terrano and Ramon Romero has many common aspects with the „traditional", also called „discount" usability testing method.
The discount usability engineering approach is based on the argument that a (relative) small number of users can identify most of the usability issues. Moreover, the identified results can be used to improve software usability [1]. Specifically, one popular theory claims that five users provide the most testing result as the law of diminishing returns apply if more users were tested.
The advocates of the RITE method observed that usability literature fail to focus on one of the most important aspects that practitioners value in a commercial environment: to deliver an improved interface as fast and cheapest possibly. Authors of the RITE method stipulates that whenever a discount usability approach is used within a project, it is more important to fix the identified usability issues rather than arguing over the aspect of identifying all usability issues that might be there.
It is important to mention that only few studies also focused on the probability of implementation for the identified usability issues. An even fewer studies covered this aspect in more depth, by analyzing the degree of improvement in the user interface of a shipped product or the relative effectiveness of these improvements affecting commercial sales or user efficiency [2]. The RITE promoters identified 4 reasons that can explain why usability issues that are uncovered do not get fixed [3]:
The decision makers are not convinced of the degree of importance of the identified issues;
Solving usability issues takes time and resources and when decision team if faced with the option to fix an issue or add a new feature, most of the times the last option is favored;
Late usability feedback, the delay between when a feature is implemented and when usability feedback is delivered to the team is a barrier to those recommendations being used;
Many times the project teams are not sure if the proposed solution will fix the issue.
The RITE method approach minimizes the above described problems related to implementation of usability issues. This method focuses on making very rapid changes to the user interface. More significantly, it evaluates the efficacy of user interface changes immediately after (sometimes within hours) of implementation.
In the HCI handbook, it is described as a method where "fewer participants are used before implementing changes, but more cycles of iteration are performed. With RITE, it is possible to run almost 2 to 3 times the total sample size of a standard usability test. However, only 1 to 3 participants are used per iteration with changes to the prototype immediately implemented before the next iteration (or group of one to three participants)" [4].
Case study: First use of RITE method
The RITE method (as described by Michael Medlock and his colleagues) was firstly used by Microsoft Game Center during the development of Age Of Empires II (AoE 2), one of the RTS (Real Time Strategy) PC games developed by the company. It was specifically used in usability testing of the "tutorial" component of the game.
Participants in the test were a group of individuals with no experience in playing RTS type of games but were interested in playing them. All of them played at least one game on the PC on the year prior to launching of AoE 2.
The metrics used during the tests were based on the observation of the participants, specifically failures (errors made that resulted in user failure to continue the tutorial) and errors (errors made that resulted in user confusion).
Is it worth mentioning that during tests, the build was changed after the first participant. The reason was that at some early point in the game tutorial the user is instructed to gather wood by chopping trees with one of its villagers. However, the development team failed to put any trees on the visible screen. The user spent a lot of time confused as waht needs to do. The problem and the solution to the problem identified were both clear: place trees on the visible screen and instruct user how to explore and find more trees off-screen. After these were done, the above described issue was never encountered again with the rest of participants. Once the „chopping trees" issue was resolved the users were able to progress further into the tutorial thus encountering other issues along the way. This illustrates an important strength of the RITE method. By fixing errors early, additional problems can be uncovered.
Another issue was related to allready fixed problems. At some point in the tutorial, the users are required to „train 5 swordsmen". The tutorial interface contained some incosistencies, in some place the interface said „train units" in others said „create units". The users thought that the swordmen can be created by moving the villagers into the barracks structure (converting villagers into swordsmen), while the units can be created only from within barracks structure itself. Noticing several participants failing to complete the tutorial task, the team decided that the terminology inconsistency was the cause of the problem and decided to use the term „train units" throught the game interface. However, after the fix, the next participant failed to complete the task and the problem reoccured in exact same way. As a result the team changed the terminology to "create unit" throughout the rest of the user interface. Once it was "re-fixed" this problem was not seen again.
Case study: SMS 2.0 - Converting SMS language into Formal language
The RITE method was used in testing the usability of the website that provides the support for the mobile application that allows automatic translation from SMS language (also known as textish or lingo) into formal language. The website contained a translation interface where the user can input SMS language text and translate it into formal language.
Participants in the test were smartphone users that use SMS when communicate with others. The group consisted of 5 users (2 males and 3 females) that did not use this service before but expressed their intention to use one.
The participants were required to evaluate a low fidelity prototype and express their feedback. The majority of participants encountered errors while trying to translate SMS language words that were not recognized by the system. The issue was identified by the team, and a dictionary was added to the website. However after the fix, one of the participants tried to translate the word "GTG" which was translated by the team as "got to go". The actual word the user tried to say was "good to go" which states a different situation for that particular user. Therefore the team decided to leave the translation to the user and added an interface that allows the user to add their own words and translations that will be added into the website dictionary after it was approved by an editor in order to avoid inappropriate content being uploaded into the dictionary.
Benefits and critics of the RITE method
The RITE method goals are to identify and address as many issues as possible in the shortest time possible.
