Annals of the American Thoracic Society

Rationale: Systematic reviews (SRs) provide the best evidence of the effectiveness of treatment strategies for asthma. Carefully conducted SRs provide high-quality evidence for supporting decision-making, but the trustworthiness of conclusions can be hampered by limitation in rigor.

Objectives: To appraise the methodological quality of a representative sample of SRs on asthma treatments in a cross-sectional study.

Methods: A cross-sectional study was conducted to identify SRs on asthma treatment published between 2013 and 2019 by searching the Cochrane Database of Systematic Reviews, Embase, MEDLINE, and PsycINFO. SRs including at least one meta-analysis on asthma treatments were included. Methodological quality of included SRs was assessed with the Assessing the Methodological Quality of Systematic Reviews 2 instrument. Factors associated with methodological quality were explored using multivariate regression analysis.

Results: One hundred thirty-six SRs were included and appraised, with a majority being non-Cochrane reviews (71.3%). Only 12 (8.8%) were of high overall quality; 9 (6.6%), 32 (23.5%), and 83 (61.0%) were of moderate, low, and critically low overall quality, respectively. More specifically, no SRs (0.0%) conducted a comprehensive literature search; only 3 (2.2%) justified why a particular primary study design was selected; 37 (27.2%) reported sources of funding among included studies; and 54 (39.7%) provided lists of excluded studies with justification. Cochrane reviews (adjusted odds ratio, 36.56; 95% confidence interval, 10.49–127.42) and SRs published after 2017 (adjusted odds ratio, 4.52; 95% confidence interval, 1.73–11.83) were positively associated with higher methodological quality.

Conclusions: Methodological quality of SRs on asthma treatments are suboptimal. Future SRs should be improved by conducting comprehensive literature searches, justifying study design selection, providing a list of excluded studies, and reporting funding sources of included studies.

Asthma is one of the major contributors to global disease burden and is the 16th leading cause of disability, leading to poor quality of life, disability, and even death (1, 2). The average prevalence of doctor-diagnosed asthma among adults aged 18–45 years was 4.3% globally, with substantial variation ranging from 0.2% in China to 21.0% in Australia (1). Using wheezing as a measure for asthma, the global prevalence among children and adolescents is much higher (3), with a prevalence reaching 11.5% among 6–7-year-old children, ranging from 6.8% in the Indian subcontinent to 21.7% in Oceania.

Currently, it is well recognized that there is no cure for asthma. Because asthma is a chronic condition, the goals of asthma management are to control symptoms, minimize future risk of exacerbations, alleviate airflow limitations, and reduce treatment-related adverse events (4). An evidence-based, customized treatment would enable patients to live fully active and normal lives (2, 5). Many treatments are available for asthma, and it is crucial for healthcare providers to make sound decisions related to their application that are based on the best available clinical evidence.

Systematic reviews (SRs) use explicit and systematic methods to identify, critically appraise, and summarize evidence from all eligible studies that meet prespecified eligibility criteria documented in protocols (6). Meta-analysis is a statistical method that is commonly used to synthesize results quantitatively in an SR, with the ability to increase the precision of effect estimation and facilitating quantitative investigation of heterogeneity among the included studies (6). A well-conducted and up-to-date SR can be a major source of high-quality, relevant, and accessible information for healthcare decision makers (6, 7).

In the past decade, the volume of SR publications has been increasing rapidly, and the use of SRs for supporting decision-making has gained popularity among clinicians (8). However, methodological quality of SRs can vary, and SRs should not always be considered good sources of evidence. Methodological limitations of an SR might lead to overestimation or underestimation of treatment effects and consequently to inappropriate decision-making (7). Users of scientific evidence, including healthcare professionals, policy makers, managers, and patients, should assess the methodological quality of SRs before accepting the results and conclusions as a basis for decision-making (9).

In the field of asthma treatment, little is known about the methodological quality of SRs. In this cross-sectional study, our aims included the following: 1) to describe the bibliographic features of a representative sample of SRs on asthma treatments, 2) to evaluate SRs’ methodological rigor using the Assessing the Methodological Quality of Systematic Reviews 2 (AMSTAR-2) tool (10), and 3) to examine the relationship between methodological quality and bibliographic characteristics. AMSTAR-2 has been successfully applied as a critical appraisal tool for SRs on polycystic ovary syndrome treatments (11) as well as for SRs on robotic surgery (12). Our aim was to provide insights into how the methodological quality of SRs on asthma treatment could be improved in the future.

Eligibility Criteria

All SRs of randomized controlled trials published after 2013 that included at least one meta-analysis on asthma treatment were eligible. SRs published after 2013 were purposefully sampled because the median duration for an SR to become outdated is 5.5 years; hence, older SRs were of less interest. The SRs had to focus on patients with asthma diagnosed on the basis of recognized criteria (e.g., Global Initiative for Asthma) (4). There were no restrictions on types of interventions and outcomes. Narrative reviews, protocols, conference abstracts, overviews of SRs, network meta-analyses, or SRs without meta-analyses were excluded. When duplicate SRs were identified, the most up-to-date versions were selected for evaluation.

Literature Search

We identified potentially eligible citations using “asthma” and related keywords (see Appendix E1 in the online supplement) to search four electronic databases: the Cochrane Database of Systematic Reviews, MEDLINE, Embase, and PsycINFO. This allowed us to create a representative sample with both Cochrane and non-Cochrane SRs. Searches of the last three databases were performed through the Ovid platform using specialized search filters for SRs (1315). Searches were limited to English or Chinese citations published between January 2013 and October 2019.

Literature Selection and Data Extraction

All retrieved citation records were screened on the basis of eligibility criteria. Details on bibliographic characteristics (e.g., year of publication, impact factor of the journal, and number of included trials) were extracted with a prepiloted data extraction form (Appendix E2).

Methodological Quality Assessment

We assessed the methodological quality of the included SRs using AMSTAR-2, a validated appraisal tool with 16 quality-related items, with 7 items being considered as critical items (10). Assessment of results from the 16 individual items allowed a rating of overall methodological quality: high quality, moderate quality, low quality, or critically low quality (10). Operational details on the AMSTAR-2 assessment framework are shown in Appendix E3. Processes of citation screening, data extraction, and methodological quality assessment were completed by two reviewers independently, with disagreements being resolved by consensus, referring to the original publication, or arbitration by a third senior investigator.

Data Analysis

Data derived from AMSTAR-2 appraisal and bibliographic characteristics are summarized as frequencies, percentages, medians, and ranges, as appropriate. Kruskal-Wallis rank tests were used to evaluate differences in the overall methodological quality across different bibliographic characteristics.

Seven bibliographic characteristics were chosen as independent variables in regression analyses to investigate their potential association with the rigor of SRs: 1) whether the SR was a Cochrane review, 2) whether it was an update of a previous SR, 3) whether it was an SR on pharmacological treatment, 4) year of publication, 5) impact factor of the journal in the year before the SR was published, 6) number of authors, and 7) location of the corresponding author. Multiordinal regression analysis was used to explore the potential association between overall methodological quality and bibliographic characteristics. Associations between bibliographic characteristics and the rating of each individual AMSTAR-2 item were further explored with binary logistic regression (for items 1, 3, 4–6, and 10–16, which used binary ratings) or multinominal logistic regression (for items 2 and 7–9, which were rated across three categories).

The adjusted odds ratio (AOR) was calculated to quantify association between AMSTAR-2 ratings and bibliographic characteristics. Model fitting was checked with the Hosmer-Lemeshow test (for binary logistic regression), likelihood ratio test (for multinomial logistic regression), and Pearson test/deviance test (for multiordinal regression). A P value less than 0.05 was considered statistically significant, except for the Hosmer-Lemeshow test, Pearson test, and deviance test, where P > 0.1 indicated adequate model fitting. All statistical analyses were conducted with IBM SPSS Statistics version 25 software (IBM Corp.).

Literature Screening and Selection

The search identified 2,242 records, of which 136 SRs met the eligibility criteria and were included (Appendix E4). Details on literature selection are reported in Figure 1.

Bibliographic Characteristics of Included SRs

The 136 SRs included 1,969 randomized controlled trials, which recruited 401,881 patients with asthma. A median of nine trials were included in each SR (range, 2–98). The median impact factor of the journals publishing the SRs was 3.7 (range, 0–47.7). More than two-thirds (71.3%) of the SRs were non-Cochrane reviews. Only 13 (9.6%) were updates of previous SRs, and, among these, the majority (12; 92.3%) were Cochrane reviews. Both pharmacological (104; 76.5%) and nonpharmacological (29; 21.3%) treatments were covered by the SRs. The highest proportion of corresponding authors was from Asia (59; 43.4%), followed by Europe (39; 28.7%). However, most funding for conducting the SRs was sourced from Europe (37; 27.2%), followed by Asia (28; 20.6%) (Table 1).

Table 1. Bibliographic characteristics of 136 included systematic reviews on asthma treatments

Bibliographical CharacteristicsResults*
Cochrane review39 (28.7)
Non-Cochrane review97 (71.3)
An update of a previous review13 (9.6)
 An update of a previous Cochrane review12 (8.8)
 An update of a previous non-Cochrane review1 (0.7)
Publication year, median (range)2016 (2013–2019)
Publication journal impact factor, median (range)3.7 (0.0–47.7)
Number of review authors, median (range)5.0 (2–25)
Location of corresponding author 
 Europe39 (28.7)
 America31 (22.8)
 Asia59 (43.4)
 Oceania7 (5.1)
Type of treatment 
 Nonpharmacological29 (21.3)
 Pharmacological104 (76.5)
 Both3 (2.2)
Total number of included primary studies1,969
Median number of included primary studies in each SR (range)9.0 (2–98)
Total number of participants included in primary studies401,881
Median number of participants included in primary studies1,491 (157–27,951)
SRs reporting intervention harms118 (86.8)
 Nonpharmacological treatments23 (79.3)
 Pharmacological treatments92 (88.5)
 Both treatments2 (66.7)
Funding location of the SR 
 Europe37 (27.2)
 America17 (12.5)
 Asia28 (20.6)
 Oceania5 (3.7)
 Not reported24 (17.6)
 No funding support25 (18.4)
SRs that searched English databases136 (100.0)
SRs that searched non-English databases39 (28.7)
Report year span of search 
 Yes, reported both starting and ending years103 (75.7)
 Partially, only reported starting years5 (3.7)
 Not mentioned28 (20.6)
Search terms reported for one or more electronic databases 
 Topics/free text/keywords/MeSH47 (34.6)
 Full Boolean72 (52.9)
 Readers are referred elsewhere for full search strategy7 (5.1)
 No research term10 (7.4)
Eligibility criteria based on language of publication 
 English and non-English101 (74.3)
 English publications only23 (16.9)
 Not reported12 (8.8)
Risk-of-bias assessment tools 
 Cochrane risk of bias110 (80.9)
 Jadad scale9 (6.6)
 Schulz approach1 (0.7)
 Delphi list2 (1.5)
 More than one tool4 (2.9)
Risk-of-bias assessment tool not used10 (7.4)
Included a PRISMA-like flow diagram134 (98.5)

Definition of abbreviations: MeSH = National Library of Medicine Medical Subject Headings; PRISMA = Preferred Reporting Items for Systematic Reviews and Meta-Analyses; SR = systematic review.

*Values are n (%) or median (range).

The percentages were calculated by using the total number of categories as the denominator.

All included SRs searched English databases; however, only 39 (28.7%) searched non-English databases. The majority (101; 74.3%) had no language restriction in their inclusion criteria, whereas 23 (16.9%) only considered English-language publications. One hundred ten (80.9%) adopted the Cochrane risk-of-bias tool for assessing the risk of bias, 9 (6.6%) used the Jadad scale, and 10 (7.4%) did not use any tool for appraising the risk of bias (Table 1).

Methodological Quality of SRs on Asthma Treatments
Overall methodological quality

Only 12 SRs (8.8%) were judged as having a high methodological quality, 9 (6.6%) were of moderate quality, 32 (23.5%) were of low quality, and the remaining 83 (61.0%) were of critically low quality (Table 2). When compared with non-Cochrane reviews, Cochrane reviews showed a higher proportion of high-quality SRs (23.1% vs. 3.1%; P < 0.001). SRs conducted by corresponding authors from Europe (23.1%) had higher proportions of high-quality SRs than their counterparts from America (0.0%), Asia (3.4%), and Oceania (14.3%) (P < 0.001). Most SRs (81.4%) with corresponding authors from Asia were of critically low methodological quality. Interestingly, all updates of SRs were of low or critically low quality (Table 2).

Table 2. Methodological quality of the 136 systematic reviews on asthma treatments

CharacteristicsHigh QualityModerate QualityLow QualityCritically Low QualityP Value
Total12 (8.8)9 (6.6)32 (23.5)83 (61.0)
Cochrane review    <0.001
 Yes9 (23.1)5 (12.8)20 (51.3)5 (12.8)
 No3 (3.1)4 (4.1)12 (12.4)78 (80.4)
An update of a previous review   <0.001
 Yes0 (0.0)0 (0.0)11 (84.6)2 (15.4)
 No12 (9.8)9 (7.3)21 (17.1)81 (65.9)
Published year    0.153
 2013–20178 (7.9)4 (4.0)24 (23.8)65 (64.4)
 2018–20194 (11.4)5 (14.3)8 (22.9)18 (51.4)
Location of corresponding author   <0.001
 Europe9 (23.1)1 (2.6)12 (30.8)17 (43.6)
 America0 (0.0)4 (12.9)11 (35.5)16 (51.6)
 Asia2 (3.4)4 (6.8)5 (8.5)48 (81.4)
 Oceania1 (14.3)0 (0.0)4 (57.1)2 (28.6)
Type of treatment    0.287
 Nonpharmacological5 (17.2)0 (0.0)7 (24.1)17 (58.6)
 Pharmacological7 (6.7)9 (8.7)25 (24.0)63 (60.6)
 Both0 (0.0)0 (0.0)0 (0.0)3 (100.0)
Reported intervention harms    0.938
 Yes11 (9.3)8 (6.8)28 (23.7)71 (60.2)
 No1 (5.6)1 (5.6)4 (22.2)12 (66.7)
Funding support    0.299
 Yes9 (10.3)7 (8.0)21 (24.1)50 (57.5)
 No1 (4.0)1 (4.0)9 (36.0)14 (56.0)
 Not reported2 (8.3)1 (4.2)2 (8.3)19 (79.2)
Searched non-English databases   0.063
 Yes2 (5.1)3 (7.7)4 (10.3)30 (76.9)
 No10 (10.3)6 (6.2)28 (28.9)53 (54.6)
Performance on each individual AMSTAR-2 criterion

The included SRs generally performed well on 7 of the 16 AMSTAR-2 items, with over 80% rated as satisfactory on the following quality criteria: 1) included PICO (problem/patient/population, intervention/indicator, comparison, outcome) components in the research questions and inclusion criteria (86.0%); 2) performed duplicate study selection (84.6%) and 3) data extraction (87.5%); 4) appropriately assessed the risk of bias of the included studies (83.8%) and 5) accounted for the risk of bias when interpreting the results (82.4%); 6) provided satisfactory explanation for the observed heterogeneity (89.0%); and 7) reported information on conflicts of interest for the SR (87.5%). However, only two of these criteria (appropriately assessed risk of bias and accounting for this risk when interpreting the results) were considered critical in the AMSTAR-2 tool.

There was much room for improvement in the following, because more than 30% of the included SRs did not fulfill these quality criteria, with the first four items being critical (Figure 2 and Appendix E5):

  1. Provided a list of excluded studies with justifications (82; 60.3%)

  2. Reported an a priori protocol and justified deviations from the protocol (64; 47.1%)

  3. Used appropriate methods to pool the results (54, 39.7%): Among these 54 SRs, the most common inappropriate meta-analysis method was the misuse of a random effects model as a “remedy” for high heterogeneity (25; 46.3%), of which sources of heterogeneity were not explored despite high I2 values (I2 ≥ 75%).

  4. Investigated publication bias (52; 38.2%)

  5. Reported sources of funding among the included studies (99; 72.8%)

  6. Assessed the potential impact of risk of bias among the included trials during meta-analysis (51; 37.5%)

Furthermore, it is noteworthy that none of the 136 SRs were rated “yes” on having performed a comprehensive literature search. The majority (84.6%) of studies were judged as “partial yes” only. In addition, a majority (97.8%) of SRs did not provide an explanation on why a certain study design should fit the eligibility criteria. Both items were critical items in the AMSTAR-2 tool.

Factors associated with methodological quality

Multiordinal regression analyses identified two factors that were significantly associated with overall methodological quality of SRs: Cochrane reviews showed better overall quality than non-Cochrane reviews (AOR, 36.56; 95% confidence interval [CI], 10.49–127.42), and more recently published SRs were more rigorous than those published before 2017 (AOR, 4.52; 95% CI, 1.73–11.83) (Table 3). Both the Pearson test (P = 0.979) and the deviance test (P = 1.000) showed good fitting of the regression models.

Table 3. Association between characteristics of asthma treatments, systematic reviews, and methodological quality: ordinal logistic regression analysis

Bibliographic Characteristics (Independent Variable)Adjusted OR (95% CI)P Value*
Cochrane review36.56 (10.49–127.42)<0.001
Year of publication4.52 (1.73–11.83)0.002
Pharmacological treatment§0.39 (0.15–1.02)0.054
Update of previous review0.29 (0.08–1.10)0.070
Impact factor1.03 (0.97–1.10)0.304
Number of review authors0.97 (0.81–1.16)0.739
Corresponding author was from America1.63 (0.52–5.17)0.405
Corresponding author was from Oceania1.84 (0.33–10.24)0.486
Corresponding author was from Asia0.99 (0.28–3.50)0.990

Definition of abbreviations: CI = confidence interval; OR = odds ratio.

*The P values of all Pearson chi-square tests and deviance were >0.1, indicating good model fit for all logistic regression analyses.

Non-Cochrane reviews were used as a reference.

Year of publication was divided into two periods, of which those published in 2013–2017 were used as a reference.

§Systematic reviews of nonpharmacological interventions were used as a reference.

Systematic reviews that are not updates of previous review were used as a reference.

Systematic reviews with corresponding author from Europe were used as a reference.

Binary logistic regression analyses indicated that, as compared with non-Cochrane reviews, Cochrane reviews showed better performance for the following four AMSTAR-2 items: performing duplicated data extraction (AOR, 28.11; 95% CI, 1.78–442.80), reporting sources of funding for the included studies (AOR, 30.48; 95% CI, 6.13–151.49), assessing the potential impact of risk of bias during meta-analysis (AOR, 5.95; 95% CI, 1.39–25.49), and investigating publication bias (AOR, 4.80; 95% CI, 1.40–16.41). SRs published after 2017 performed better in including PICO components in the research questions and inclusion criteria (AOR, 1.43; 95% CI, 1.01–2.04). SRs published in journals with higher impact factors were more likely to conduct duplicate study selection (AOR, 1.56; 95% CI, 1.07–2.27). As compared with SRs conducted by teams with a European corresponding author, those completed by teams with Asian corresponding authors showed better performance in duplicate data extraction (AOR, 8.20; 95% CI, 1.52–44.16) (Table 4). All multinominal logistic regressions showed poor model fitting.

Table 4. Association between characteristics of asthma treatments, systematic reviews, and methodological quality of individual AMSTAR-2 item: Logistic regression analysis

AMSTAR-2 Item (Dependent Variables)PredictorsAOR (95% CI)P Value*
1. Did the research questions and inclusion criteria for the review include the components of PICO?Published more recently1.43 (1.01–2.04)0.047
5. Did the review authors perform study selection in duplicate?Higher impact factor1.56 (1.07–2.27)0.021
6. Did the review authors perform data extraction in duplicate?Corresponding author was from Asia8.20 (1.52–44.16)0.014
 Cochrane review28.11 (1.78–442.80)0.018
10. Did the review authors report on the sources of funding for the studies included in the review?Cochrane review30.48 (6.13–151.49)<0.001
12. If meta-analysis was performed, did the review authors assess the potential impact of RoB in individual studies on the results of the meta-analysis or other evidence synthesis?Cochrane review5.95 (1.39–25.49)0.016
 Pharmacological intervention§3.02 (1.15–7.94)0.025
15. If they performed quantitative synthesis, did the review authors carry out an adequate investigation of publication bias (small study bias) and discuss its likely impact on the results of the review?Cochrane review4.80 (1.40–16.41)0.012

Definition of abbreviations: AMSTAR-2 = Assessing the Methodological Quality of Systematic Reviews 2; AOR = adjusted odds ratio; CI = confidence interval; PICO = population, intervention, control group, and outcome; RoB = Cochrane tool for assessing risk of bias in randomized trials.

*The P values of all Hosmer-Lemeshow tests were >0.1, indicating good model fit for all logistic regression analyses.

Systematic reviews with corresponding authors from Europe were used as a reference.

Non-Cochrane reviews were used as a reference.

§Systematic reviews of nonpharmacological interventions were used as a reference.

This cross-sectional study identified and appraised 136 SRs on asthma treatments published between 2013 and 2019, covering both pharmacological and nonpharmacological interventions, as well as Cochrane and non-Cochrane reviews. Although Cochrane reviews generally had better overall methodological quality, the rigor of most of the included SRs was not satisfactory. Only 8.8% SRs were of high quality; 6.6% were of moderate quality; 23.5% were of low quality; and 61% were of critically low quality. Much improvement is needed in providing an a priori protocol of the review, justifying the selection of the study design, conducting a comprehensive literature search, providing lists of excluded studies, and reporting funding sources for the included studies, because less than 40% of SRs scored “yes” for these items. Nevertheless, some improvement over time was observed, with SRs published more recently showing better methodological quality.

Providing A Priori Protocols and Lists of Excluded Studies

As compared with narrative reviews, SRs are considered to be more objective (6). One reason for this is that SRs preestablish a review method in the protocol, and justification is required for any deviation from the protocol. Furthermore, primary studies are less likely to be excluded because of unfavored results, based on the requirement of documenting a list of excluded studies with justifications for exclusion (6, 10). The requirements of an a priori protocol and a list of excluded studies are compulsory for Cochrane reviews but not for the majority of other journals’ reviews (6). Consequently, only 35.3% and 39.0% of the 136 SRs provided an a priori protocol with justifications for any deviation and a list of excluded studies, respectively. It is advised that authors of future non-Cochrane reviews register their protocol in PROSPERO (international prospective register of systematic reviews) (16), report deviations from the protocol together with justifications, and provide a list of excluded studies with justifications for exclusion as an appendix when confronting word limitations of the targeted journal.

Conducting Comprehensive Literature Searches to Reduce Publication Bias

A comprehensive literature search is a predeterminant for identifying an unbiased sample of SRs that answer the same research question, regardless of the publication status. It has been demonstrated that studies with positive results have a higher chance of being published with disregard of their scientific rigor (17). This addresses the importance of putting effort into identifying unpublished research. AMSTAR-2 sets a stricter criterion for comprehensive literature searches by including a reference list check, searching gray literature, consulting experts, and keeping the search updated (10). On the basis of the updated criteria, none of the included 136 SRs conducted a comprehensive literature search. In addition, bias may also result from search restriction on English-language publications only, because positive results had a higher probability of being published in English-language journals (18). Our results showed that less than one-third of the SRs searched non–English-language databases. Furthermore, 38.2% of the SRs did not consider the risk of publication bias and its potential influence on the results. Special attention should be given to performing a comprehensive literature search, preventing publication bias, and assessing such risk in future SRs, because this is a very common shortcoming for SRs across different medical conditions (1921).

Assessing Risk of Bias with Appropriate Tools

It is acknowledged that risk of bias among included primary studies is critical for the trustworthiness of the evidence derived from SRs (22). Assessing the risk of bias of included studies by means of an appropriate tool is a key component of an SR. Although the majority (83.8%) of sampled SRs assessed risk of bias among included studies, it is worth noting that 6.6% used the Jadad scale, which is an outdated tool that lacks assessment of important risk-of-bias domains such as allocation concealment (23). Moreover, 7.4% of the SRs did not provide any information on their critical appraisal results. SR authors, journal editors, and reviewers should pay attention to this issue to ensure the quality of future SRs.

Declaring Conflicts of Interest for Both the SR and the Included Studies

Funding support, particularly from commercial sources, might introduce bias into study results by drawing conclusions favoring the supported interventions and is commonly observed in pharmacological interventions (21). Conflicts of interest in an SR generally originate from two potential sources: the included primary studies and the SR itself. These two sources have been distinguished as two separate items in AMSTAR-2 (item 10 and item 16) (10). The SRs appraised in this study generally performed well in stating conflicts of interest of the SR itself (87.5% rated “yes”). However, further improvements are needed to declare the same information for primary studies included in the SR, because less than one-third were rated as “yes” in this aspect. Moreover, 76.5% of the included SRs summarized evidence from pharmacological treatments, which in turn may further jeopardize the trustworthiness of these SRs (24).

Factors Related to the Methodological Quality of SRs

Regression analyses of individual items showed that, as compared with SRs with corresponding authors from Europe, those with corresponding authors from Asia performed better in terms of duplicate data extraction (item 6). However, this result was not replicated in regression analysis for overall quality, which might be explained by the fact that item 6 is not considered as critical in AMSTAR-2 (10). Some previous publications have found that Asian authors have poorer performance on individual AMSTAR items (20, 25), whereas others found conflicting results (26), and yet other studies suggested that methodological quality is not related to authors’ region of origin (12, 27, 28). Inconsistencies among different studies might be explained by variation in medical conditions focused on and how regions of origin were classified (12). Future evidence users should not judge the quality of SRs based solely on authors’ region of origin without a formal critical appraisal process.

Our results demonstrated that SRs published more recently have better methodological quality, resonating with observations from previous publications (12, 21). This encouraging trend may be explained by successful popularization of rigorous SR methodology promoted by the Cochrane handbook (6) as well as AMSTAR-2 (10). It is also possible that, as authors accumulate more experience with conducting and publishing SRs, their methodological skills also improve over time. It is currently recommended that an experienced systematic reviewer is indispensable for a successful team (29), and it would be interesting to investigate how systematic reviewers’ contributions would actually impact methodological quality in the future.

Comparisons with Similar Studies

Although our results demonstrated that SRs of asthma treatments had better performance (8.8% of high quality) than SRs on robotic surgery (12), SRs with high methodological quality remained rare. Similar results have also been observed in SRs on other medical conditions, including acupuncture for polycystic ovary syndrome (11), depression treatments (21), and low-carbohydrate diets for overweight and obesity (30). These consistent observations signal an urgent need for improving methodological quality of SRs.

When considering performance of individual AMSTAR-2 critical items, our sample showed better performance than other SRs in providing an a priori protocol and lists of excluded studies with justifications, assessing the risk of bias among included studies appropriately, and accounting for this risk when interpreting the results for robotic surgery (12), low-carbohydrate diets for overweight and obesity (30), and acupuncture for polycystic ovary syndrome (11). However, our sample was inferior to these three studies in terms of performing a comprehensive literature search (11, 12, 30).

Strengths and Limitations

This study had two strengths. First, using AMSTAR-2, we assessed the most up-to-date sample of SRs on asthma treatments (10). AMSTAR-2 has been shown to be a validated appraisal tool and has been widely applied in other methodological research (11, 12, 30, 31). Second, beyond overall methodological quality, we also reported detailed performance on each individual AMSTAR-2 item to provide information on specific areas that require improvement.

Several limitations of this study should be noted. First, we only included SRs published between 2013 and 2019. Thus, performance of SRs published before 2013 was not evaluated. Second, only SRs published in the English or Chinese language were included, which may reduce the representativeness of our sample. Third, our assessments were based only on published information and hence appraisal results may be limited by underreporting, which is a common shortcoming of this type of study (12).


SRs with meta-analyses have been considered one of the best sources of evidence, given their ability to increase statistical power, improve the precision of effect estimation, and critically appraise primary studies (6, 32). However, flaws in the design and conduct of SRs may yield biased results (10, 33), which may, in turn, impact the outcomes of clinical practice if decisions are made on the basis of biased results. For instance, an evaluation on the impact of including gray literature in SRs demonstrated that failure to incorporate unpublished data would lead to significant underestimation or overestimation of treatment effect, leading to underuse of effective treatments or vice versa (34). Indeed, it is not uncommon to find SRs with inferior methodological quality (1921, 26), which suggests that healthcare providers, guideline developers, and other stakeholders should critically appraise SRs when considering adoption of their results (10). More important, as gatekeepers of the scientific rigor of publications, journal editors and peer reviewers are strongly suggested to evaluate the quality of SR submissions using AMSTAR-2 to ensure that only SRs with reasonable rigor are published.


The overall methodological quality of SRs on asthma treatments is suboptimal, with only 8.8% judged as having a high overall quality. Cochrane reviews and SRs that were published more recently had better overall quality. Increased efforts are needed to provide an a priori research design through protocol registration, to develop a list of excluded studies with justifications, to conduct a comprehensive literature search, to justify the selection of the study design, and to report sources of funding of all included primary studies. SR authors, journal editors, and peer reviewers should strive to improve methodological quality of SRs by adhering to the requirements of AMSTAR-2 and PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines.

1 . To T, Stanojevic S, Moores G, Gershon AS, Bateman ED, Cruz AA, et al. Global asthma prevalence in adults: findings from the cross-sectional world health survey. BMC Public Health 2012;12:204.
2 . Global Athsma Network. The global asthma report 2018. Auckland, New Zealand: Global Asthma Network; 2018 [accessed 2019 Nov 2]. Available from:
3 . Lai CKW, Beasley R, Crane J, Foliaki S, Shah J, Weiland S; International Study of Asthma and Allergies in Childhood Phase Three Study Group. Global variation in the prevalence and severity of asthma symptoms: phase three of the International Study of Asthma and Allergies in Childhood (ISAAC). Thorax 2009;64:476483.
4 . Global Initiative for Asthma. Global strategy for asthma management and prevention (2018 update). 2018 [accessed 2019 Nov 2]. Available from:
5 . World Health Organization. World Health Organization fact sheet on asthma. 2017 [accessed 2019 Nov 2]. Available from:
6 . Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al., editors. Cochrane handbook for systematic reviews of interventions, 2nd ed. Chichester, UK: John Wiley & Sons; 2019.
7 . Mulrow CD, Cook DJ, Davidoff F. Systematic reviews: critical links in the great chain of evidence. Ann Intern Med 1997;126:389391.
8 . Moher D, Tetzlaff J, Tricco AC, Sampson M, Altman DG. Epidemiology and reporting characteristics of systematic reviews. PLoS Med 2007;4:e78.
9 . Fleming PS, Koletsi D, Seehra J, Pandis N. Systematic reviews published in higher impact clinical journals were of higher quality. J Clin Epidemiol 2014;67:754759.
10 . Shea BJ, Reeves BC, Wells G, Thuku M, Hamel C, Moran J, et al. AMSTAR 2: a critical appraisal tool for systematic reviews that include randomised or non-randomised studies of healthcare interventions, or both. BMJ 2017;358:j4008.
11 . Luo YN, Zheng QH, Liu ZB, Zhang FR, Chen Y, Li Y. Methodological and reporting quality evaluation of systematic reviews on acupuncture in women with polycystic ovarian syndrome: a systematic review. Complement Ther Clin Pract 2018;33:197203.
12 . Yan P, Yao L, Li H, Zhang M, Xun Y, Li M, et al. The methodological quality of robotic surgical meta-analyses needed to be improved: a cross-sectional study. J Clin Epidemiol 2019;109:2029.
13 . Montori VM, Wilczynski NL, Morgan D, Haynes RB; Hedges Team. Optimal search strategies for retrieving systematic reviews from Medline: analytical survey. BMJ 2005;330:68.
14 . Wong SSL, Wilczynski NL, Haynes RB. Developing optimal search strategies for detecting clinically sound treatment studies in EMBASE. J Med Libr Assoc 2006;94:4147.
15 . Health Information Research Unit, Evidence-Based Health Informatics. Search strategies for PsycINFO in Ovid syntax. Hamilton, ON, Canada: McMaster University; 2016 [accessed 2019 Oct 10]. Available from:
16 . Booth A, Clarke M, Dooley G, Ghersi D, Moher D, Petticrew M, et al. The nuts and bolts of PROSPERO: an international prospective register of systematic reviews. Syst Rev 2012;1:2.
17 . Easterbrook PJ, Berlin JA, Gopalan R, Matthews DR. Publication bias in clinical research. Lancet 1991;337:867872.
18 . Egger M, Zellweger-Zähner T, Schneider M, Junker C, Lengeler C, Antes G. Language bias in randomised controlled trials published in English and German. Lancet 1997;350:326329.
19 . Ho RS, Wu X, Yuan J, Liu S, Lai X, Wong SY, et al. Methodological quality of meta-analyses on treatments for chronic obstructive pulmonary disease: a cross-sectional study using the AMSTAR (Assessing the Methodological Quality of Systematic Reviews) tool. NPJ Prim Care Respir Med 2015;25:14102.
20 . Wu XY, Lam VC, Yu YF, Ho RS, Feng Y, Wong CH, et al. Epidemiological characteristics and methodological quality of meta-analyses on diabetes mellitus treatment: a systematic review. Eur J Endocrinol 2016;175:353360.
21 . Chung VCH, Wu XY, Feng Y, Ho RST, Wong SYS, Threapleton D. Methodological quality of systematic reviews on treatments for depression: a cross-sectional study. Epidemiol Psychiatr Sci 2018;27:619627.
22 . Sterne JAC, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ 2019;366:l4898.
23 . Jadad AR, Moore RA, Carroll D, Jenkinson C, Reynolds DJM, Gavaghan DJ, et al. Assessing the quality of reports of randomized clinical trials: is blinding necessary? Control Clin Trials 1996;17:112.
24 . Chopra SS. MSJAMA: industry funding of clinical trials: benefit or bias? JAMA 2003;290:113114.
25 . Zhang H, Han J, Zhu YB, Lau WY, Schwartz ME, Xie GQ, et al. Reporting and methodological qualities of published surgical meta-analyses. J Clin Epidemiol 2016;70:416.
26 . Wu XY, Du XJ, Ho RS, Lee CC, Yip BH, Wong MC, et al. Characteristics and methodological quality of meta-analyses on hypertension treatments—a cross-sectional study. J Clin Hypertens (Greenwich) 2017;19:137142.
27 . Papageorgiou SN, Papadopoulos MA, Athanasiou AE. Evaluation of methodology and quality characteristics of systematic reviews in orthodontics. Orthod Craniofac Res 2011;14:116137.
28 . Tian J, Zhang J, Ge L, Yang K, Song F. The methodological and reporting quality of systematic reviews from China and the USA are similar. J Clin Epidemiol 2017;85:5058.
29 . Møller AM, Myles PS. What makes a good systematic review and meta-analysis? Br J Anaesth 2016;117:428430.
30 . Churuangsuk C, Kherouf M, Combet E, Lean M. Low-carbohydrate diets for overweight and obesity: a systematic review of the systematic reviews. Obes Rev 2018;19:17001718.
31 . Moore PA, Ziegler KM, Lipman RD, Aminoshariae A, Carrasco-Labra A, Mariotti A. Benefits and harms associated with analgesic medications used in the management of acute dental pain: an overview of systematic reviews. J Am Dent Assoc 2018;149:256265, e3.
32 . Mulrow CD. Rationale for systematic reviews. BMJ 1994;309:597599.
33 . Shea BJ, Grimshaw JM, Wells GA, Boers M, Andersson N, Hamel C, et al. Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews. BMC Med Res Methodol 2007;7:10.
34 . Hart B, Lundh A, Bero L. Effect of reporting bias on meta-analyses of drug trials: reanalysis of meta-analyses. BMJ 2012;344:d7202.
Correspondence and requests for reprints should be addressed to Charlene H. L. Wong, Ph.D., Jockey Club School of Public Health and Primary Care, Prince of Wales Hospital, Shatin, Hong Kong 999077. E-mail: .

Supported by the High-level Talents Introduction Plan from Central South University (no. 502045003) and the National Natural Science Foundation of China (no. 81973709).

Author Contributions: I.X.Y.W., C.H.L.W., and V.C.H.C. conceived and designed the research question. Y.D., H.W., and Y.C. extracted data. H.W. analyzed the data. I.X.Y.W. and Y.D. interpreted the results. I.X.Y.W. prepared the manuscript. C.H.L.W. and V.C.H.C. did the proofreading. All authors had full access to all of the data (including statistical reports and tables) in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. C.H.L.W. is the guarantor.

This article has an online supplement, which is accessible from this issue’s table of contents at

Author disclosures are available with the text of this article at


No related items
Comments Post a Comment

New User Registration

Not Yet Registered?
Benefits of Registration Include:
 •  A Unique User Profile that will allow you to manage your current subscriptions (including online access)
 •  The ability to create favorites lists down to the article level
 •  The ability to customize email alerts to receive specific notifications about the topics you care most about and special offers
Annals of the American Thoracic Society

Click to see any corrections or updates and to confirm this is the authentic version of record