A systematic review is "[a] review of a clearly formulated question that uses systematic and explicit methods to identify, select, and critically appraise relevant research, and to collect and analyse data from the studies that are included in the review. Statistical methods (meta-analysis) may or may not be used to analyse and summarise the results of the included studies" (Greens & Higgins, 2005). SRs are claimed to be the best source of evidence in clinical practice and decision-making (Cook et al, 1997). They provide summaries of evidence from a myriad of primary studies which focus on the same questions (Cook et al, 1997, Sánchez-Meca & Botello, 2010) by effectively managing and integrating considerably large amount of existing information (Mulrow, 1994). An overview of available scientific evidence which addresses a specific problem make time consuming process of reading individual studies unnecessary and thus, help health care professionals save their precious time (Sánchez-Meca & Botello, 2010). By summarising results of included research studies into a single statement, SRs provide greater advantage to clinicians in assessing evidence (Stevens, 2001). Furthermore, SRs resolve inconsistencies of studies that discuss the same problem but yield confusing and conflicting results (Stevens, 2001, Sánchez-Meca & Botello, 2010). Besides, SRs establish generalisability by assessing whether clinical findings are consistent across populations and settings or vary according to particular subsets (Mulrow, 1994, Stevens, 2001).
SRs have become increasingly vital to a broad range of stakeholders (Moher et al, 2007), particularly health care providers, researchers and decision makers (Mulrow, 1994). Health care providers especially clinicians read SRs to keep abreast with their specialty (Swingler et al, 2003, Moher et al, 2007) and to remain educated in wider aspects of medicine (Mulrow, 1994). Health policy makers and clinical guideline developers use SRs as starting point in formulating clinical guidelines and legislations (Mulrow, 1994, Moher et al, 2007). As for some medical journals, SRs are crucial as they serve as prerequisite evidence base tools to justify the need to conduct further research (Young & Houltan, 2005).
2.2 Overall reporting quality of SRs
Over the past few decades, SRs are being published annually in increasingly large numbers (Shea et al, 2002). A study conducted by Moher et al (2007) showed that there are about 2500 SRs indexed annually on Medline. However, there is comparatively little existing data on the reporting quality of SRs despite number of SRs published is enormous (Shea et al, 2002).
Several earlier studies concluded that quality of reporting of SRs was generally poor (Sacks et al, 1987. Mulrow et al, 1987, Silagy, 1993, Mc. Alister et al, 1999). Sacks et al (1987) evaluated the reporting quality of 86 meta-analyses of reports of randomised controlled trials published in English language by taking into consideration 23 items covering six essential domains, i.e. "study design, combinability, control of bias, statistical analysis, sensitivity analysis and application of results". The results of the study showed that reporting was generally poor, where only 24 of 86 meta-analyses (28%) addressed all six domains and of the 23 items, between 1 and 14 were satisfactorily reported (mean = 7.7, standard deviation = 2.7) (Sacks et al, 1987).
Another earlier evaluation of SRs by Mulrow et al (1987) examined 50 reviews published between June 1985 and June 1986 in 4 major medical journals and found that no single review met all eight explicit criteria of which the assessment was based on, i.e. "purpose, data identification, data selection, validity assessment, quantitative synthesis, quality synthesis, summary and future directives" (Mulrow et al, 1987). An update of this study involving 158 reviews published in six general medical journals in 1996 noted little improvement with only 2 reviews met all 10 methodological criteria and the median number of criteria fulfilled was one (Mc. Alister et al, 1999).
Silagy (1993) evaluated 28 reviews covering a wide range of subject areas which were published in seven main primary care journals in 1991 based on eight criteria. The results of the study showed that only one quarter of the reviews scored 8 points out of 16 points (2 points allocated for each clearly reported criterion, 1 point for each not clearly reported criterion and 0 point for unreported criterion) (Silagy et al, 1993).
More recently, a study by Jadad et al (1998) concluded that Cochrane reviews have superior "methodological rigor" and are more regularly updated compared with SRs or meta-analyses published in paper-based journals.
Oslen et al (2001) assessed the quality of Cochrane reviews and noted that in general, there were no problems or only trivial problems found in most of the reviews. They studied 53 reviews published in issue 4 of the Cochrane Library in 1998 and found that major problems were identified in 15 reviews (29%), which correspond to the conclusion not fully supported by the evidence in 9 reviews (17%), inadequate reporting in 12 reviews (23%) and "stylistic problems" were recognized in 12 reviews (23%) (Oslen et al, 2001).
Moher et al (2007) examined the epidemiology and reporting characteristics of 300 SRs indexed in Medline during November 2004 and found that great differences exist between Cochrane reviews and non-Cochrane reviews in the reporting quality of several characteristics. Main aspects of SR methodology were not reported in many non-Cochrane reviews, for instance, only 11% of the reviews mentioned working from a protocol in the process of completing the review. Besides, data obtained from the study suggested that the quality of reporting is inconsistent.
2.3 SRs on herbal medicines for mental and behavioural disorders
2.3.1 St John's wort (Hypericum perforatum) for depression
Hypericum extracts have been studied and included in clinical trials since the 1980s (Linde et al, 2009). Several systematic reviews published from 1995 to 2008 concluded that hypericum extracts are more effective compared with placebo and comparable to (similarly effective as) standard antidepressants in treating depressive disorders (Linde et al, 1996, Kim et al, 1999, Gaster & Holroyd, 2000, Williams et al, 2000, Whiskey et al, 2001, Linde et al, 2005, Clement et al, 2006, Linde et al, 2008). However, some of the trials included in a few reviews (Linde et al, 1996, Kim et al, 1999, Gaster & Holroyd, 2000, Williams et al, 2000) were being criticised because they incorporated patients with very few and/or mild symptoms who did not meet the inclusion criteria of major depression, were carried out by primary care physicians who were lack of experience in depression research, and/or used low doses of comparator drugs (Shelton et al, 2001).
Linde et al (2005) conducted an update of previously completed review (Linde et al, 1996) by including several new well-designed placebo-controlled trials where negative findings were found in some of the trials (Shelton et al, 2001). The results obtained provoked new debates on the efficacy of hypericum extracts for treatment of depression and the analyses showed that effects of hypericum extracts over placebo were less pronounced in studies restricted to patients with major depression (Linde et al, 2005). In order to minimise clinical heterogeneity as well as to reveal the fact that almost all new high-quality trials of hypericum extracts are restricted to patients with major depression, another update of review (Linde et al, 2008) was conducted by including several new well-designed trials restricted to patients with major depression. 29 trials were included in the study. In nine larger trials and nine smaller trials involving comparison of hypericum extract with placebo, the combined response rate ratio (RR) obtained was 1.28 (95% confidence interval (CI), 1.10-1.49) and 1.87 (95% CI, 1.22-2.87) respectively. As for comparison with standard antidepressants, RRs were 1.02 (95% CI, 0.90-1.15; 5 trials) for tri- or tetracyclic antidepressants and 1.00 (95% CI, 0.90-1.11; 12 trials) for selective serotonin reuptake inhibitors (SSRIs). Hence, it can be concluded that hypericum extracts tested in the included trial are more effective than placebo and are similarly effective as standard antidepressants in patients with major depression (Linde et al, 2008).
2.3.2 Anxiety
There are SRs on three herbal medicines, i.e. kava extract, valerian and passiflora for anxiety (Pittler & Ernst, 2003, Miyasaka et al, 2006, Miyasaka et al, 2007). Only study conducted by Pittler & Ernst (2003) found that kava extract is more effective than placebo in symptomatic treatment of anxiety despite the size of the effect is small. Finding of SR carried by Miyasaka et al (2006) comparing the effectiveness of valerian with placebo and diazepam for anxiety showed that there is no significant differences between valerian and placebo and between valerian and diazepam in Hamilton Anxiety (HAM-A) total scores. Besides, a definite conclusion was unable to be drawn as there was only a single small study with 36 patients available (Miyasaka et al, 2006). Miyasaka et al (2007) conducted a SR by including 2 studies to compare the effectiveness of passiflora with benzodiazepines, i.e. mexazolam and oxazolam respectively in treating anxiety. None of the study was able to distinguish passiflora from benzodiazepines in any of the outcome measures. Two possible reasons for this lack of statistical difference identified were the medications were equally effective and insufficient number of studies were included (sample size was not large enough) (Miyasaka et al, 2007).
2.3.3 Dementia
SRs on three herbal medicines, i.e. Zhiling decoction, Yizhi capsule and Huperzine A for vascular dementia found no convincing evidence to support the use or effectiveness of these herbs (Jirong et al, 2004, Wu et al, 2007, Hao et al, 2009). Jirong et al (2004) found no suitable randomised placebo-controlled trials and concluded that the available evidence was inadequate to support the use of Zhiling decoction in the management of vascular dementia. Wu et al (2007) conducted a SR of Yizhi capsule for vascular dementia found no study that met the inclusion criteria and no evidence from randomised controlled trials to assess the potential of Yizhi capsule in treating vascular dementia.
SR conducted by Birks & Grimley Evans (2009) to assess the efficacy of Gingko biloba for dementia included 36 trials but most were small and the duration was less than 3 months. More recent trials with longer duration showed inconsistent results for cognition and activities of daily living when comparing Gingko biloba with placebo and 1 of the trials reported large treatment effects in favour of Gingko biloba (Birks & Grimley Evans, 2009). Another SR conducted by Weinmann et al (2010) which included 9 trials with 2372 patients found that Gingko biloba appeared to be more effective than placebo for dementia. Data obtained showed statistical significant advantage of Gingko biloba compared with placebo in improving cognition as well as statistical significant advantage of Gingko biloba compared with placebo in improving activities of daily living in subgroup of patients with Alzheimer's disease. Results for quality of life and neuropsychiatric signs and symptoms were inconsistent (Weinmann et al, 2010).
2.3.4 Schizophrenia
A SR conducted by Rathbone et al (2005) to review Chinese herbal medicine, either being used alone or as a part of Traditional Chinese Medicine (TCM) approach for people with schizophrenia found that Chinese herbal medicines, given in a Western biomedical context, may be beneficial for people with schizophrenia when combined with antipsychotics.
2.3.5 Insomnia/Sleep disorders
2 of the SRs and/or meta-analyses of valerian for insomnia yield inconclusive evidence of the benefit of valerian as a sleep aid (Stevinson & Ernst, 2000, Bent et al, 2006, Taibi et al, 2007). This was because the included studies of these reviews presented great inconsistency across patients, experimental designs, procedures and methodological quality (Stevinson & Ernst, 2000). As for review conducted by Bent et al (2006), the studies included showed great heterogeneity in terms of doses, preparations and length of treatment. Taibi et al (2007) conducted a SR to examine the evidence of valerian for insomnia and found that overall evidence did not support the clinical efficacy of valerian as a sleep aid. A meta-analysis including 18 randomised controlled trials was conducted by Fernández-San-Martín et al (2010) and the qualitative dichotomous results showed that valerian was effective for subjective improvement of insomnia. However, the effectiveness of valerian was not demonstrated with quantitative measurements (Fernández-San-Martín et al, 2010).
2.4 Characteristics and Quality of SRs of herbal medicines
According to Linde et al (2003), "descriptive empirical studies" on SRs are relatively uncommon. Linde et al (2003) conducted a research to study the characteristics and quality of SRs on acupuncture, herbal medicines and homeopathy by including 115 SRs with 58 SRs on herbal medicines. The characteristics and quality of the included SRs examined are summarised in the tables below (Table 2.1 & 2.2).
Table 2.1 Characteristics of included SRs (Linde et al, 2003)
Characteristic
E.g.(s)
Bibliographic characteristics
Year of publication
Question
Narrow intervention focus
Condition reviewed
Psychiatric
Information on inclusion criteria
Explicit inclusion criteria regarding patients/condition; important inclusion criteria, e.g. only placebo-controlled trials
Literature search
Explicitly in Medline
Others
Methods; results and conclusion
Table 2.2 Quality of included SRs (Linde et al, 2003)
Items reviewed
Search methods reported
Comprehensive search
Inclusion criteria reported
Selection bias avoided
Validity criteria reported
Methods for combining reported
Findings combined appropriately
Conclusions supported by data
Some limitations encountered in the study were discussed, for example, limitations in resources causing half of the reviews were extracted and assessed by only 1 reviewer and there was great heterogeneity across some of the included reviews. Therefore, it was suggested that the analysis of the data only served to give an overall view of the descriptive epidemiology of available SRs on herbal medicines and there is still plenty room for improvement in future SRs conducted on herbal medicines (Linde et al, 2003).
2.5 Guidance/appraisal tools to evaluate reporting quality of SRs
The increasing popularity and usefulness of SRs urged the reports of SRs to be "clear, accurate and transparent" (Moher, 2008). Despite there are some improvement in the reporting of SRs, the quality of reporting is still inconsistent (Moher et al, 2007). Therefore, it is of paramount importance to follow reporting standard or reporting checklist (Wiesler & McGauran, 2010).
QUOROM (QUality Of Reporting Of Meta-analyses) statement, which serves as a standard to enhance the reporting quality of "meta-analyses of randomised controlled trials (RCTs)" was developed in 1996. QUOROM checklist consists of a total of 20 headings and subheadings and describes the preferred ways of reporting of meta-analyses in terms of abstract, introduction, methods, results and discussion (Moher et al, 1999).
Table 2.3 Quality of reporting of meta-analyses
Heading
Subheading
Descriptor*
Reported? (Y/N)
Page number
Title
Abstract
There are 6 items, i.e. objectives, data sources, review methods, results and conclusion
Introduction
Methods
There are 6 items, i.e. searching, selection, validity assessment, data abstraction, study characteristics and quantitative data synthesis.
Results
There are 3 items, i.e. trial flow, study characteristics and quantitative data synthesis
Discussion
*Detailed descriptor please refers to Moher et al (1999).
Source: Moher et al, 1999
QUOROM statement was revised and renamed PRISMA (Preferred Reporting Items for Systematic reviews and Meta-Analyses) statement (Moher et al, 2009). PRISMA checklist consists of 27 items (see Appendix 5 for detailed checklist items). PRISMA checklist differs from QUOROM checklist in a few aspects as shown by the table below (Table 2.4).
Table 2.4 Substantive specific changes between the QUOROM checklist and the PRISMA checklist (a tick indicates the presence of the topic in QUOROM or PRISMA)
Section/topic and item
QUOROM
PRISMA
Comment*
Abstract
Introduction:
Objective
Methods:
Protocol
Search
Assessment of risk of bias in included studies
Assessment of risk of bias across studies
Discussion
Funding
*Detailed comment please refers to Moher et al (2009).
Source: Moher et al, 2009
