Researchers have often tried to improve the usability, portability, integrity and other aspects of a software in order for it to be more user friendly and gain user trust. In order to achieve this, software systems are becoming increasingly larger and more complex than before. The large software size and complexity has contributed significantly to the delay in applications development and causes software developers to face many issues during system development. In recent years, the rapid growth of the information sphere has become the most important issue in the world. Human lives have become almost impossible without computers and people have come to require computing to solve their daily problems and needs. Several models, approaches, and techniques have been proposed to reduce the negative effects of software size and complexity. The aim of this study is to analyze the field of software quality and examine previous research on it, in order to better understand the state of the art, and to consider certain steps for its improvement. The emphasis in this study will be on reviewing major publications in computer science journals from the year 2008 to 2009. The existing software quality classification schemes appear to be insufficient to express the desired dimensions in software quality. Therefore, a multi-faceted classification system that characterizes software quality researches along several dimensions is needed. This study shows the new classification dimensions on software quality research. It also shows the current state of software quality research from the methods and approaches used to conduct software quality research, the source of data used in experimental software quality studies, the research approaches and methods used to formulate software quality models, and the basics of proposed models in software quality researches, as well as the domain of these proposed models.
INTRODUCTION
The rapid growth of information in recent years and the increase of user demand has resulted in a need for systems of larger size and complexity. Several researchers have focused their research on the issues of reducing software complexity and minimizing the efforts required for building large systems. Hence, many software quality factors have been identified to solve those issues in which maintenance is the most important and complex factor. Many unsolved simple mistakes that may occur during the early stages of software development may grow to be a big fault at a later stage. Therefore, maintenance is the most complex and costly facet in a software life cycle. More importantly, some cost estimation researches that have been done during recent years have yielded very interesting results regarding the cost of software maintenance, such as the increase of maintenance cost from 52% of overall software cost in 1995 to 76% in 2005 (Jones, 2006). IEEE estimated that the annual cost of software maintenance in the United States exceeds $70 billion (Edelstein, 1993). Other researchers have estimated the range of software maintenance to be between 40% and 80 % of the total cost of one software lifecycle (Alkhatib, 1992; Green, 2000).
Several papers that examine the nature of research within computer science tend to analyze the field of computing (see, for example, Glass et al., 2004), while others focus on specific areas of computer science (see, for example, Gruman, 1990; Rice, 1995; Wegner and Doyle, 1996; Gallopoulous and Sameh, 1997) or even subareas, for example, heterogeneous databases (Sheth and Larson, 1990), or data modeling (Hull and King, 1987), rather than on the discipline as a whole. It was found that many studies focused on the field of computer science at a national level ,for example, Estivili-Castro (1995) on Maxico and Ramamritham (1997) on India, while others focused on the leading scholars and institutions of computing (see, for example, Top scholars and institutions in the software engineering field 1998-2002 Glass and Chen, 2003)
The objective of this study is to provide a detailed characterization of software quality research, along the dimensions identified above, by examining articles published in major computer science journals in 2008 and 2009. The interest here goes beyond research methods and approaches and includes other ways in which the research is conducted, which includes identifying the object studied and the results of the theoretical foundation of the research. To achieve such goals, the following questions were asked and their answers solidified.
What research approaches do Software Quality researchers use?
What research methods do Software Quality researchers use?
Which type of research approaches are used to formulate a software quality model?
What solutions do Software Quality researchers propose?
How do the Software Quality researchers propose their models?
What types of software are the proposed models able to evaluate?
What is the relationship between research approaches and methods used to formulate software quality models?
The analytical portion of this research began by defining general classification systems to cover approaches and methodologies used to solve the software quality issues. The classification system was based on the scheme developed by Glass et al., 2004 and Cheon et al., 1993.
The current study
This section describes the classification scheme used to characterize Software Quality research. It also presents details regarding the journals examined and the classification process.
Classification scheme
In this study, characterizing research in software quality, the first task was to identify a classification scheme that would enable the encompassing of all the richness of the software quality research field. It was found that traditional classification schemes such as the ACM computing classification scheme (ACM CCS, 1998) characterizes research only along one dimension i.e., topic. Classification schemes in related disciplines such as in information systems, e.g., ISRL categories (Barki et al., 1988), also tend to focus on research topic with research method as a secondary consideration. Ramamritham (1997) classified the research on various computing disciplines in India while Estivili-Castro (1995) did a similar study on Mexico. Other classifications that have been developed include a multi-faceted classification system (Ramesh et al., 2004; Glass et al., 2004) that characterizes research along several dimensions focusing on general topics of computing disciplines. Still others have focused on specific topics or even sub areas in computing disciplines, for example Glass and Chen (2003) that classified the leading scholars and institutions of the various computing disciplines. Sheth and Larson (1990) worked on heterogeneous databases whereas Hall and King (1987) focused on data modeling.
Because this analysis focuses solely on software quality research, it excludes many of the categories and sub-categories that were classified by previous researchers which are not relevant to this subject. For example, since Glass et al. (2004) found that action research, ethnography, instrument development, and protocol analysis studies constituted 0% of both the computer science and software engineering fields. Hence, we are not considering them in this research. The focus here is on other studies with specific interest in subjects such as the formulation of approaches, which was found by Glass et al. (2004) to constitute 55.3% of software engineering studies.
Because none of the existing classification schemes were sufficiently rich in the desired dimension, it was necessary to develop a more specific classification scheme in software quality. Hence, a classification system that is comprehensive and accurate in a wider perspective was developed to describe research in software quality. Though, some categories in this scheme are more specific towards software quality, other categories are general to the computing discipline. For brevity, the classification system is presented with the results of this study in seven tables; table 2 to table 8. Below, we present a brief description of how the classification systems were developed.
Research approach
The choice of research approaches narrows the set of possible applicable research methods. In this analysis Morrison and George's (1995) system is used as a basis for classification. This system divides the research approaches into four main categories, which are descriptive, evaluative, formulative, and developmental. The combination of the developmental and descriptive categories under the descriptive category is so done because such research primarily involves describing systems (Ramesh et al., 2004). For example, Anda et al. (2009) used the evaluative approach to assess the changes in the utilization of large software project case models for his analysis. Kumar et al. (2009) used the formulative model approach to evaluate the quality of Aspect-Oriented Software, and Wagner et al.(2008) used the formulative approach to find a way to evaluate and assess quality requirements in a structured and comprehensive way.
Research method
Research method describes the specific techniques used in a given study. Unlike Glass et al.'s (2004) classification that included 19 categories, this paper excluded some of those categories which were considered irrelevant while others categories were reshaped in order to produce four main categories. In this analysis, the main categories related to this area used by (Glass et al., 2004) were adopted. The papers in this analysis are classified into four categories that are closely related to software quality. The categories used are descriptive, experimental, literature review, and survey. For example, Hassine et al. (2010) used a survey method to provide readers with sufficient knowledge about existing timed scenario approaches to guide them in making informed decisions to when and how the time aspects can be incorporated in their development process. On the other hand, Molina and Toval (2009) used the experimental method to reduce some of the quality failures detected in Web Engineering development projects.
Formulative research approaches
In this category we classified the formulative research approaches used in software quality researches. The classification was based on Glass et al. (2004). For a more detailed perspective, this study includes the formulative approach, separate formulative process and formulative method. It also removes the classes of formulative concept, formulative standards, formulative algorithms, and formulative taxonomy because none of these approaches were found in the selected studies. After modifying the classification scheme, seven categories were finally included in this study, which were the formulative approach, formulative classification, formulative framework, formulative guideline, formulative method, formulative model, and formulative process.
Source of data
Source of data classification explains the resources used by the experimental studies included herein. This study found that all the experimental studies used data and applications. Hence, both are sources of information. It was found that all the selected papers used three ways in selecting their data which are real world data/applications, laboratory data, and case studies. The real world data allows the researcher to show the side effects of the proposed solution in real situation that may not be faced it in the lab, such as Behkamal et al. (2009) who used the ISACO portal from the Iranian Khodro Company. The laboratory data which is included simple case studies, allows the researcher to find the best case that can illustrate the proposed solution, since it is difficult to find real world cases that include all the factors that a researcher looks for in order to explain a solution. For example, Molina and Toval (2009) used the example of an online book sale system and assumed that the CEO of a certain company had decided to expand her/his business strategy, and that her/his goal is to attract new clients. The third method, which involved using data from other studies or participants (case study), involved the researchers selecting their data or applications from previous research which included real and laboratory applications. This allowed them to compare their results with previous research in order to show the efficiency of their results. For example, Zheng (2009) collected his datasets from other studies to be used in his experiment. He then compared the performance of his system with the previous study to show the contribution of his study.
Domain of a Model
The domain of model is defined as the range of applicability a model proposes. In this case, the domains of proposed models are categorized according to the type of software products evaluated by the said models. It was found that these models are distributed into five categories, which are general models that are proposed to evaluate general software quality products, e-commerce models proposed specifically to evaluate e-commerce software, object oriented model for object oriented aspects, management models which are used to evaluate teamwork productivity, and educational models that used to evaluate educational software products. For example, Gashi et al. (2009) proposed a software quality model to evaluate the COTS components, and Fang (2008) proposed a software quality model to evaluate educational software.
Basis of Model
The 'basis of model' classification scheme illustrates how the software quality models are formulated and addresses the models that researchers use as the basis to formulate new models. This basis is classified into four main categories which are the ISO model, new model, combination of models, and modifications from other models. The models based on modified or customized ISO models are suitable for specific types of software products. New models that were proposed depended on the factors identified basically for specific problem. Modifications from other model were models that were improved or enhanced from previous models in order to be more efficient. Categorization based on ISO standards, which is part of the modified model category, was identified as an independent category because many researchers used it in their papers except those who combined it with other models. For example Loh and Lee (2009) combined five software quality models to propose their model; these models were the McCall Model, Boehm Model, FURPS/FURPS+, ISO 9126, and Dromey Model. This study was identified as being under the category of combination of models not based on the ISO. Behkamal et al. (2009) made a comparison between common software quality models and the ISO 1926 model. The gaps in the ISO model that made it unable to evaluate B2B applications were identified, and finally the ISO model was modified to produce a new software quality model able to evaluate B2B applications. Therefore, this study categorizes it into the category "based on ISO", but not under combination of models category.
Methods used to formulate software quality models
This study categorizes the methods that were used to formulate software quality models. In this classification, Glass et al's., 2004 research method was used. It was found that the methods used in the papers were distributed into three categories. These methods are-experimental and captured papers whose models are evaluated by case studies; descriptive-which includes the papers that only describe their proposed models or describe other models to identify the gaps that need to be solved; and survey, which is used by experts to evaluate their proposed models. For example, Singh et al. (2009) proposed the Support Vector Machine (SVM) model to find the relationship between object-oriented metrics, which was then evaluated by using an open source dataset.
Journal and article selection
In this study, to truly reflect the field of software quality, there is a need to ensure that a wide range of sample research articles are represented. To begin with, the ACM and IEEE journals that Geist et al. (1990) used in their study of faculty productivity, Ramesh et al. (2004) used in their study of computer science, Glass and Chen (2003) used in their study on top scholars and institutions in software engineering, and the papers used by Glass et al. (2004) in computing disciplines were included. In addition to that, other journals in the Science direct and Springer repositories were selected for a thorough view.
A simple approach was used to facilitate in the selection of 70 articles for evaluation. This was to ensure that the selected articles focused directly on software quality issues and were equally reflected in the sample based on their keywords. Furthermore, in-depth looks into their abstracts were made to ensure its close relation to software quality. Hence, there were almost an equal number of articles from each of the selected journals. This approach resulted in 22 articles from the IEEE journals, 18 articles from the ACM journals, 22 articles from ScienceDirect journals, and 8 articles from Springer journals. Table 1 represents the journals which include the selected articles.
Findings
The research finding addresses the research approaches, research methods, domain of models, basis of models, and the research methods and approaches used to formulate a models that software quality researchers have been observed to utilize. Tables 2 to 9 summarize the findings.
Research approach
The most common research approach used to solve the problems of software quality is the formulative approach, which constitutes 76% of all the samples. The second most popular method is the evaluative approach, which stands at 16%, while the descriptive approach has the lowest usage at only 8%.
Figure 1 Research Approach
Research method
Most studies that have been done on software quality used the experimental method to evaluate their results, with 47% of the sampled studies utilizing this method. The second most utilized method, which made up 36% of the studies was the descriptive method, which is used to describe previously proposed solutions and discussed these solutions to come out with newer effective solutions. The third is the survey method which is used to identify the problems and evaluate the solutions from other points of view such as experts and users. This method constituted 10% of the survey. The least utilized method, which makes up 7% of the survey, is the review of Literature which was used to identify the definitions and formulate classifications.
Figure 2 Research Method
Formulative research approaches
This study focuses on this category because it was the most utilized approach in software quality researches, as shown previously. We classified all studies according to Glass et al.'s (2004) classification scheme. The most utilized approaches in software quality researches were the formulative approach, 30% and formulative model, 28%. Other methods used were the formulative process, 13%, formulative method 11%, formulative framework 8%, formulative classification 6%, and formulative guideline 4%.
Figure 3 formulative research approach
Source of data
From the studies that used the experimental methods, it was found that most of them select their data from other research (case studies), 43%, while the others obtained their data from real world applications, 36%, and laboratory cases, 21%.
Figure 4 data source for experimental studies
Model domain
From the studies that propose models to solve quality issues, most models were specific for certain purposes while a few of these models were proposed for general purposes. A classification of these models based on the model objective is done by grouping them into five categories. These categories include OO and AO evaluation models, which constitutes 26%, Team management and information system evaluation models at 27%, e-commerce and COTS evaluation models at 20%, general purpose models used to evaluate any type of software products at 20%, and educational software evaluation with 7%.
Figure 5 domain of model
The basis of a model
This category illustrates the bases used by researchers to formulate their models. 33% of them formulated their models based on previous models proposed, 33% used a combination of a number of proposed models, 27% based their models on ISO model, and 7% based on the software product factors.
Figure 6 Basis of model
Methods used to formulate software quality models
This category illustrates the methods used to build software quality models. From the papers that proposed software quality models, it was found that the three commonly used methods were the experimental method 46%, used to evaluate proposed models practically for more efficiency. The descriptive methods are 47%, which is used to explain the strengths and weaknesses in the previous models and the combinations between them to develop a new model. Survey is 7%, which is used with other points of view about the proposed model such as expert opinion and user feedback.
Figure 7 Methods used to formulate software quality models
Findings for methods and approaches used to build software quality researches
The approaches used with methods may help to identify the suitable method and approach combination for specific issues in software quality.
Figure 8 Research methods and Approaches Used in Software Quality Researches
Discussion and implications
In this study, we sought to analyze the characteristics of software quality research according to eight research categories, some of which are recorded in literature as being important aspects of any research study, while we propose the others specifically for software quality research. A brief summary of the key findings is first of all provided followed by a discussion of some of the limitations of this study.
Classification schemes were primarily based on previous classifications, and then an in-depth classification of software quality researches according to analytical parameters was carried out. The proposed categories gave detailed information about the software quality researches that were conducted in 2008 and 2009.
Software quality research is distributed across three major approaches: descriptive, evaluative, and formulative. The leading category is the formulative approach, while few papers fall in the descriptive and evaluative categories. The focus here is on the formulative approach because of the number of papers found under this category made up 75.71% of all papers reviewed. Classification of this category into seven sub-categories was carried out, which leads to the study of formulative approaches and formulative models.
Since researchers focused on formulative models, analysis of this category was done in two ways. The first dealt with how researchers proposed their models. This was done by grouping sample papers into four sub-categories. The second dealt with which type of software the proposed model was able to evaluate and this included grouping papers into five categories. For more details, the methods used to formulate software quality model was presented. It was found to be fairly distributed between experimental and descriptive methods and only one paper used the survey method.
In terms of research methods, software quality research was distributed across four major methods, which was lead by the experimental and descriptive categories, with very few papers falling into the literature and survey categories. In the research methods study, experimental category in terms of type of experiment is focused which classified into three sub-categories: real world cases, simulated cases, and cases obtained from other studies.
The study of this nature is not without limitations, the first limitation stems from the choice of journals. The result of this study reflects the nature of software quality research to the extent that the journals are representative of the field. The second potential limitation arises from the fact that only a sample of the studies published in the selected journals are coded. Given, however, that a systematic sampling procedure is used, there is no reason to believe that the results are biased.
Conclusion
In this study, 70 papers published in 2008 and 2009 in 26 leading research journals and conference proceedings in the computer science field were examined and software quality researches in terms of the research approaches, research methods, domain of proposed model, basic of model, type of research approach used in software quality studies, data and applications used to conduct experimental studies in software quality researches, and methods and approach used in software quality model formulation were characterized.
The results from this study will be of value to both researchers and doctoral students engaged in software quality research. For example, the study provides a characterization of research methods and approaches used in formulating software quality models and the basis of software quality models formulation. It is also beneficial to software developers and professionals who are presented with the common research models used, the common ways used to develop software quality models, and the interests of software quality researchers.
Further, this study presents the domains of interest of selected researches, which allows identification of the gaps in software quality research.
Researchers in future can use this method to analyze other subjects in software engineering or computer science. They can analyze the software quality research from other points of view.
