Beyond generally taking less time and money to conduct, observational studies have a number of strengths, as outlined in Table 1 (12). They have the potential to provide high-quality evidence. They can complement evidence from efficacy and pragmatic RCTs when concerns exist about the representativeness of patient and provider experience or conditions under which the trial was conducted (e.g., intervention rigidly implemented or study cohort not representative of disease populations). Observational studies have the ability to incorporate experiences of vast numbers of patients treated across a wide range of real-world practice settings, including academic and community settings. Observational studies can also be used to assess interventions that would be unethical to study with an RCT because of lack of equipoise or prevailing restrictions and attitudes toward the risk of harms and/or potential benefits. Furthermore, they can address questions that are not under the control of the investigator, such as questions about genetic markers. Finally, the large sample sizes and extended follow-up periods typical of many observational studies provide the time and statistical power needed to identify rarer exposures or outcomes—like adverse effects or later-emerging benefits/harms that could affect clinical practice guideline recommendations.

Observational studies also have limitations that are important to recognize (Table 1). First, without randomization, comparison groups—including those who have and have not received an intervention—are likely to differ in ways that are associated with the outcomes of interest. Although methods exist that can adjust for observed differences between treated and untreated individuals, it is challenging for them (although not impossible through methods such as instrumental variable analysis and Mendelian randomization) to account for unobserved differences or unmeasured confounders (12, 13). Observational studies may also select patients for study entry or exposure classification in ways that cause spurious effect estimates, even in the absence of confounding. Some common examples of biases that are not due to confounding include collider-stratification bias induced by study entry based on an effect of the exposure of interest, immortal time bias arising from improper classification of time before exposure, and unintentional conditioning on effect mediators (14, 15). Challenges to identifying, measuring, and adjusting for all potential confounders and accounting for all selection biases make observational studies more susceptible to bias than RCT designs. Second, many large observational studies rely on secondary data that were not collected for research purposes. Such data may lack desirable granular clinical detail—such as pulmonary-function and other test-result data—that would help identify patient characteristics or patient-reported outcomes, such as symptoms and quality of life. This could make these types of studies vulnerable to misclassification (if the investigators tried to categorize a variable that was not reliably measured because of lack of detail), unmeasured confounding, and conclusions based on outcomes that are not the most relevant to patients.

Observational studies have strengths and weaknesses that tend to complement those of RCTs. For example—as discussed above—whereas high-quality observational studies tend to have potential for stronger external validity, RCTs tend to have stronger internal validity. Other complementary features are shown in Table 1. Thus, the best medical evidence on which to base clinical decisions is that which is determined to be of the highest quality in both designs.

Because active participation is not required, and a waiver of informed consent can be obtained from research ethics boards, observational studies are more likely to include representative and diverse patient populations. With larger and more representative samples, observational studies provide an opportunity to explore treatment effect heterogeneity at multiple levels, including by sex, race/ethnicity, comorbidity, adherence to therapy, and access to care. Such information is fundamental to understand and overcome factors that contribute to health inequity.

In contrast to observational studies, RCTs often include participants who are not representative of the populations affected by the diseases they examine. Studies have shown that approximately 6% of patients with asthma (16) and 27% of patients with chronic obstructive pulmonary disease (17, 18) meet eligibility criteria of contemporary RCTs in these areas. A recent systematic review found that African Americans—a group disproportionately at risk of asthma morbidity and mortality—were underrepresented in RCTs assessing adherence to asthma medications (19). Barriers that prevent people from being recruited and participating in RCTs, such as strict eligibility criteria (20–22), difficulties contacting potential participants or their surrogate decision-makers, large participant burden in terms of time and effort, beliefs about research, and need for informed consent (23), tend to disproportionately impact minorities, patients of lower socioeconomic status, and patients with mental health conditions or other comorbidities (24). Having limited access to healthcare providers who participate in clinical trials and structural racism and research abuses that have led people to, understandably, distrust the medical establishment also reduce the involvement of underrepresented groups in research (25). Although we must strive to overcome such obstacles in the conduct of RCTs, observational studies may overcome such barriers to participation by underrepresented groups.

Weaknesses of observational studies may be mitigated with rigorous study design and the use of causal diagrams. Observational study methods that reduce confounding and strengthen causal inference have developed greatly in the past 15 years and can be conducted by knowledgeable researchers (12, 26, 27). One of these is targeted trial emulation, which is the application of design principles from randomized trials to the design and analysis of observational studies. This has been shown to help researchers identify and avoid unnecessary biases and provide a clear means for articulating the trade-offs that need to be made in observational studies (28, 29). In addition, reports indicate that outcomes of high-quality observational studies and RCTs often do not differ (30–32). When assessing study quality, the individual merits of each observational study should be considered, rather than discounting studies simply because of their observational nature. Tools for assessing the quality of observational studies—including those that investigate causal inference (26)—include the Newcastle-Ottawa Scale and ROBINS-I (Risk of Bias in Non-Randomized Studies–of Interventions) tool. The Newcastle-Ottawa Scale was developed to assess the quality of observational studies (http://www.ohri.ca/programs/clinical_epidemiology/oxford.asp), and the ROBINS-I tool was developed to address the risk of bias in observational studies. Other guides also exist.

Although observational studies are becoming better understood and more accepted by the scientific community, there is still some mistrust of their validity by clinicians that can lead to exclusion or discounting of their results during evidence syntheses and clinical decision-making (33). Such distrust has been perpetuated by generalizations that, although they might apply to some observational studies, are taken by some to be absolute. Table 2 outlines some of the perceptions surrounding observational studies.

Pragmatic RCTs occupy a space between traditional efficacy RCTs and observational studies. This trial design prioritizes design decisions such that study results may more closely mimic those observed in routine clinical conditions (34). Such design decisions include eligibility criteria that rely on data collected during routine care, research embedded in clinical practice, and some flexibility in intervention delivery. Although pragmatic RCTs attempt to reflect real-world circumstances, studies employing pragmatic trial designs often require informed consent and other forms of active patient participation (e.g., completing study questionnaires) that could limit the applicability of study results to real-world clinical populations. Observational studies can also evaluate interventions in real-world conditions. When done well, because they can include more people, they have the potential to address external validity in a way that pragmatic RCTs cannot.

According to the National Academy of Medicine, clinical practice guidelines make recommendations informed by the best available evidence identified by systematic reviews (35). It is imperative that clinical guidelines that influence the care of millions of people reflect evidence of best-known care. Appropriate inclusion of observational studies in evidence syntheses can contribute to this.

Currently, many guideline groups use a stepwise approach to identify the best evidence. RCTs are initially sought and, as a group, assessed for quality; if RCTs are identified and are determined to be of “sufficient” or “adequate” quality, they are used to inform a recommendation. What constitutes sufficient quality is left to the discretion of the guideline panel. If no RCTs are identified or if the available RCTs are determined to be of insufficient quality, then observational studies are sought. Observational studies of sufficient quality are first used to inform the recommendation. If no observational studies are identified or if the available observational studies are determined to be of insufficient quality, then indirect evidence is sought. Finally, if sufficient indirect evidence does not exist, then either no recommendation is made or uncontrolled studies or expert opinions (i.e., clinical knowledge and experience) are used to inform recommendations. This stepwise approach, which saves time and effort, means that observational studies are considered only on an as-needed basis, rather than being considered routinely.

Although this “as-needed” approach to the inclusion of observational studies in practice guidelines is practical and efficient, it means that the totality of evidence is often not used to inform patient care. Varying approaches used by other guideline developers may lead to the selection of different studies, resulting in inconsistent estimates of effects and different recommendations across guidelines. A more unified, widely accepted approach to the use of observational studies is desirable.

In 2005, the ATS adopted the GRADE (36) approach, a dynamic paradigm for appraising and summarizing evidence, as well as for formulating, writing, and grading recommendations (37). GRADE is also endorsed by the World Health Organization, Cochrane Collaboration, American College of Physicians, and other guideline-developing organizations (http://gradeworkinggroup.org). GRADE recognizes that study type is not the only indicator of study quality, as all study designs can be of variable quality. GRADE uses study design to make an initial assumption about the quality of evidence and then provides criteria that warrant upgrading the quality of a body of evidence (e.g., large magnitude of effect, dose–response gradient, plausible confounders contributing to opposite effect) or downgrading the quality of a body of evidence (e.g., risk of bias, indirectness, inconsistency, imprecision, and publication bias).

The GRADE working group is developing guidance for the use of observational studies in the development of clinical practice guidelines (10, 38, 39). An algorithm for including observational studies in the development of guidelines is provided in Figure 1.

Figure 1. Algorithm for including observational research in medical decision-making. GRADE = Grading of Recommendations Assessment, Development and Evaluation.

Download Figure | Download Powerpoint

[More] [Minimize]

An observational study can provide higher-quality evidence than an RCT. When no RCTs are identified or RCTs are appraised as being of low or very low quality, observational studies are sought; in this case, observational evidence is considered “replacement” evidence because it may substitute for RCT evidence or its quality may surpass RCT evidence (38). When moderate-quality RCT evidence exists, the justification for the moderate-quality rating should be reviewed. Observational studies may be sought when concerns such as indirectness, imprecision, or inconsistency have led to downgrading the RCT evidence from being of high quality to being of moderate quality; in this instance, observational evidence is considered “complementary” because it provides additional information.

Indirectness refers to situations in which the studies included patient populations, interventions, comparators, or outcomes that were different from those in the question posed by the guideline panel (36). Indirectness is often used to refer to a lack of generalizability. As an example, if a guideline panel asks about vaccinations in the elderly, but all relevant studies enrolled younger volunteers, indirectness of the population exists. Imprecision indicates that the confidence interval (CI) of the estimated effect is too wide to definitively answer the question asked by the guideline panel (i.e., the ends of the CI would lead to different clinical decisions). As an example, if a guideline panel decided a priori that a 5% mortality reduction is necessary to use a drug and the studies estimate that the drug reduces mortality by 7% with a 95% CI of 3–11%, then imprecision exists because at one end of the CI you would use the drug and at the other end you would not. Inconsistency exists if there is variability in the direction or magnitude of effect across studies; this determination may be subjective or may use the I² statistic or a P value for heterogeneity. Indirectness, imprecision, and inconsistency are the causes of downgrading evidence from having high quality to having moderate quality that warrant seeking complementary evidence because these are the limitations that may be overcome by observational evidence. As examples, consider the following: if RCTs are limited by indirectness, then observational studies that directly address the guideline question may be found; if RCTs are limited by imprecision, then large observational studies with narrow CIs may be found; and if RCTs are limited by inconsistency, then multiple consistent observational studies may be found. In contrast—although this is controversial—it is less certain that observational studies can overcome RCTs with risk of bias because observational studies, according to GRADE, also have a risk of bias.

Finally, observational studies may be sought when a guideline panel surmises that RCTs do not provide the best evidence for outcomes that the guideline committee considers essential for decision-making, such as when rare or long-term outcomes are judged as critical. In this situation, observational evidence is considered “sequential” because necessary information is not available from RCTs, so it must be obtained from observational studies. Sequential evidence and replacement evidence are frequently confused. Generally speaking, replacement evidence is from observational studies that are sought because there is no RCT evidence or very poor RCT evidence, whereas sequential evidence is sought because, although adequate quality RCT evidence exists, the RCT evidence may be incomplete or too narrow to be informative. As an example, if a guideline committee was addressing a bronchoscopic intervention for asthma, there might be high- or moderate-quality RCTs reporting short-term outcomes, but the guideline committee might also be interested in long-term outcomes not reported by the RCTs and, therefore, might seek observational studies as sequential evidence.

Observational evidence is unnecessary when RCT evidence that examines outcomes that the guideline committee considers essential for decision-making (i.e., critical outcomes) is appraised as being of high quality (i.e., observational evidence is not needed to replace, complement, or provide sequential evidence).

Our recommendations on when to integrate observational study evidence in clinical guidelines and evidence synthesis should be evaluated, and updates should be made as we learn about its benefits (e.g., how often observational evidence changes guideline recommendations). In these evaluations, consideration should be given to the evolution of the GRADE approach as well as to the added resources needed to search for and identify observational studies, review and assess their quality, and make decisions about their suitability for inclusion.