Comments
Abstract
Rationale: Early warning system (EWS) scores are used by hospital care teams to recognize early signs of clinical deterioration and trigger more intensive care.
Objective: To systematically review the evidence on the ability of early warning system scores to predict a patient’s risk of clinical deterioration and the impact of early warning system implementation on health outcomes and resource utilization.
Methods: We searched the MEDLINE, CINAHL, and Cochrane Central Register of Controlled Trials databases through May 2014. We included English-language studies of early warning system scores used with adults admitted to medical or surgical wards. We abstracted study characteristics, including population, setting, sample size, duration, and criteria used for early warning system scoring. For predictive ability, the primary outcomes were modeled for discrimination on 48-hour mortality, cardiac arrest, or pulmonary arrest. Outcomes for the impact of early warning system implementation included 30-day mortality, cardiovascular events, use of vasopressors, respiratory failure, days on ventilator, and resource utilization. We assessed study quality using a modified Quality in Prognosis Studies assessment tool where applicable.
Measurements and Main Results: Of 11,183 citations studies reviewed, one controlled trial and 20 observational studies of 13 unique models met our inclusion criteria. In eight studies, researchers addressed the predictive ability of early warning system tools and found a strong predictive value for death (area under the receiver operating characteristic curve [AUROC], 0.88–0.93) and cardiac arrest (AUROC, 0.74–0.86) within 48 hours. In 13 studies (one controlled trial and 12 pre–post observational studies), researchers addressed the impact on health outcomes and resource utilization and had mixed results. The one controlled trial was of good quality, and the researchers found no difference in mortality, transfers to the ICU, or length of hospital stay. The pre–post designs of the remaining studies have significant methodological limitations, resulting in insufficient evidence to draw conclusions.
Conclusions: Early warning system scores perform well for prediction of cardiac arrest and death within 48 hours, although the impact on health outcomes and resource utilization remains uncertain, owing to methodological limitations. Efforts to assess performance and effectiveness more rigorously will be needed as early warning system use becomes more widespread.
Observational studies suggest that patients often show signs of clinical deterioration up to 24 hours prior to a serious clinical event requiring intensive interventions (1). Early warning system (EWS) scores are used by hospital care teams to recognize early signs of clinical deterioration and trigger more intensive care, such as increased nursing attention, informing the care provider, or activating a rapid response team (RRT) or medical emergency team (2). The use of EWSs has increased since the Institute for Healthcare Improvement launched its 100,000 Lives Campaign in 2004, in which the use of RRTs was recommended to save lives (3). Many hospital systems have started to implement the use of EWS scores, and, although evidence shows that this leads to an increase in the use of RRTs and admissions to the intensive care unit (ICU), uncertainty remains as to their ability to predict an impending health event and whether they are useful in reducing cardiac arrest and death (1). In this study, we assessed the ability of EWS tools to predict cardiac arrest, pulmonary arrest, or death outcomes within 48 hours of data collection, as well as to determine the impact of EWS use on in-hospital health outcomes and utilization of resources.
We developed an analytic framework (Appendix A) with input from key informants (clinicians, nurses, hospital administration members, and patient advocates) to visually display the key questions and scope parameters of the review. We searched the MEDLINE, CINAHL, and the Cochrane Central Register of Controlled Trials databases from their inception to May 2014 for English-language studies of EWS in adult medical and surgical ward populations (Appendix B). We obtained additional articles from systematic reviews and reference lists and by consulting experts. All citations were imported into an electronic database (EndNote X4; Thomson Reuters, New York, NY).
Investigators reviewed titles and abstracts of citations identified through literature searches, and eligible full-text articles were independently assessed by two reviewers based on predefined eligibility criteria (Appendix C). Disagreements were resolved through group consensus.
To assess the predictive ability of EWSs, we examined observational studies in which researchers reported associations between EWS scores and 48-hour mortality, cardiac arrest, or pulmonary arrest. To assess the impact of EWS implementation, we included studies in which researchers reported on 30 day mortality, cardiovascular events (cardiac arrest, acute coronary syndrome, and cardiogenic shock), use of vasopressors, number of days on a ventilator, respiratory failure, and utilization of resources, including ICU admission, use of RRT, and length of hospital stay. We used a best evidence approach to guide inclusion of studies (4).
Data on population characteristics, study setting, number of subjects, number of subjects lost to full analysis, name and elements of scoring system, comparator, harms, implementation characteristics (e.g., staff training, pilot phase), and funding were abstracted from each study by one investigator and confirmed by a second investigator. For studies addressing the question of predictive ability, we abstracted data on model discrimination for outcomes of mortality, cardiac arrest, and pulmonary arrest within 48 hours. For all other studies, we abstracted patient health outcomes and resource utilization.
For studies addressing predictive ability, we adapted criteria described in the Quality in Prognosis Studies assessment tool (5). In this article, we discuss the strengths and limitations of the studies and describe the overall potential for bias as applicable. For trials addressing health outcomes, we used criteria and methods defined in the methods guide developed by the Agency for Healthcare Research and Quality (6). We synthesized data qualitatively for the pre–post observational studies because objective criteria are lacking to quantitatively rate or measure the quality in these types of studies.
From among 11,183 titles and abstracts, 202 articles were selected for full-text review (Figure 1). Of these, we included 21 studies, eight providing primary data on predictive value of EWS scores and 13 pertaining to the impact of EWS interventions. We found 13 unique models ranging from four to 16 items with scores based on aggregate weighted systems (Table 1).
| Study, country | Parameters, name of scoring system | Parameters used in the scoring system | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Heart rate | Respiratory rate | SBP | Temperature | Decreased urinary output | O2 sat | Difficulty breathing | Increase in supplemental O2 | Mental Status (LOC) | Concern | Other, specify | ||
| Rothschild et al., 2010 (10), USA | Single items, not combined | x | x | x | x | x | x | – | x | x | x | DBP, seizures, uncontrolled bleeding, change in color |
| Churpek et al., 2012 (12), USA | 4-item CART | x | x | – | – | – | – | – | – | – | – | DBP, age |
| Maupin et al., 2009 (20), USA | 5-item MEWS | x | x | x | x | – | – | – | – | x | – | – |
| Jones et al., 2011 (19), UK | Patientrack EWS | x | x | x | x | – | – | – | – | x | – | – |
| Subbe et al., 2003 (26), UK | 5-item MEWS | x | x | x | x | – | – | – | – | x | – | – |
| Churpek et al., 2012 (7), USA | 5-item MEWS | x | x | x | x | – | – | – | – | x | – | – |
| O’Dell et al., 2002 (23), UK | 5-item MEWS | x | x | x | – | x | – | – | – | x | – | – |
| De Meester et al., 2013 (17), Belgium | 6-item MEWS | x | x | x | x | – | x | – | – | x | – | – |
| Smith and Oakey, 2006 (25), UK | 6-item EWS | x | x | x | x | x | – | – | – | x | – | – |
| Patel et al., 2011 (24), UK | 6-item MEWS | x | x | x | x | x | – | – | – | x | – | Indwelling Foley catheter |
| Kellett and Kim, 2012 (8), Canada | 6-item VIEWS | x | x | x | x | – | x | – | x | – | – | – |
| Bailey et al., 2013 (15), USA | 7-item EWS | – | x | x | – | – | x | – | – | – | – | Shock index, DBP, anticoagulation use |
| Hammond et al., 2013 (27), Australia | 7-item MEWS | x | x | x | x | x | – | – | x | – | – | DBP |
| Mitchell et al., 2010 (21) | 7-item MEWS | x | x | x | x | x | x | – | – | x | – | – |
| Moon et al., 2011 (22), UK | 7-item MEWS | x | x | x | x | x | x | – | – | x | – | – |
| Green and Williams, 2006 (18), Australia | 7-item clinical marker tool | x | x | x | – | x | x | x | – | – | x | – |
| Smith et al., 2013 (11), UK | 7-item NEWS | x | x | x | x | – | x | – | x | x | – | – |
| Opio et al., 2013 (13), Uganda | 7-item VIEWS | x | x | x | x | – | x | – | x | x | – | – |
| Prytherch et al., 2010 (9), UK | 7-item VIEWS | x | x | x | x | – | x | – | x | x | – | – |
| Albert and Huesman 2011 (16), USA | 12-item MEWS | x | x | x | x | x | x | x | x | x | x | WBC count, new focal weakness |
| Churpek et al., 2014 (14), USA | 16-item EWS | x | x | – | x | – | x | – | x | x | – | Time, prior ICU stay, age, DBP, BUN, potassium, anion gap, WBC, Hb, platelet count |
Eight observational studies (six prospective cohort and two case–control) met our inclusion criteria (Table 2) (7–14). One study considered single predictors, and seven studies reported the predictive values of six distinct models of EWS scores for the outcomes of interest (death and cardiac arrest within 48 hours of measurement). No study reported on the predictive ability of EWS for respiratory arrest. Most of the models ranged from four to seven items, all of which included heart rate, respiratory rate, and blood pressure, and most included temperature and mental status. The studies were conducted in several countries (four in the United States, two in the United Kingdom, one in Canada, and one in Uganda); most were carried out in urban academic hospitals.
| Study, design, setting | Population, mean age (yr), sex | Comparison | Number of patients | Predictive measures of mortality occurring within 48 h of EWS data collection/analysis | Predictive measures of cardiac arrest occurring within 48 h of EWS data collection/analysis |
|---|---|---|---|---|---|
| Churpek et al., 2012 (12) | Medical/surgical patients, including telemetry patients | Patients without a cardiac arrest or ICU Txf | Total n = 47,427 | NR | AUROC: 0.84 (95% CI, NR) vs. 0.78 for MEWS |
| Retrospective cohort study USA | Age | n = 88 (cardiac arrest) | |||
| Academic tertiary care hospital, 500 beds | 64 (cardiac arrest) | n = 2,820 (ICU Txf) | |||
| 60 (ICU Txf) | n = 44,519 (control) | ||||
| 54 (control) | |||||
| Males | |||||
| 43% (cardiac arrest) | |||||
| 52% (ICU Txf) | |||||
| 43% (control) | |||||
| Churpek et al., 2012 (7) | Medical/surgical patients | Patients without cardiac arrest | Total n = 440 | NR | MEWS AUC (95% CI) |
| Case–control study, USA | Age | n = 88 (cases) | 0.77 (0.71–0.82) | ||
| Academic tertiary care hospital, 500 beds | 64 (cases) | n = 352 (controls) | |||
| 58 (controls) | |||||
| Males | |||||
| 43% (cases) | |||||
| 49% (controls) | |||||
| Kellett and Kim 2012 (8) | Medical (non-ICU)/surgical inpatients | None | Total n = 75,419 | 48-h mortality AUROC (95% CI) | NR |
| Retrospective cohort, Canada | Age 63 | n = 43,693 (medical) | All: 0.93 (0.91–0.95) | ||
| Urban academic hospital, 375 beds | 48.9% male | n = 30,485 (surgical) | Surgical: 0.89 (0.78–1.0) | ||
| Medical: 0.89 (0.85–0.92) | |||||
| Opio et al., 2013 (13) | Medical patients | Patients without an event | Total n = 844 | Death within 24 h | NR |
| Retrospective cohort, Uganda | Age 45.2 | AUROC (95% CI): | |||
| Hospital, no ICU care, 330 beds | 42% males | 0.89 (0.82–0.95) | |||
| Prytherch et al., 2010 (9) | General medicine and emergency patients (consecutively admitted) | Patients alive at 24 h following observation | Total n = 35,585 patient episodes | Death within 24 h | NR |
| Prospective cohort study UK | Age 67.7 | AUROC (95% CI): | |||
| Urban hospital | 47.5% males | 0.89 (0.88–0.89) | |||
| Rothschild et al., 2010 (10) | Medical and medicine subspecialty inpatients | Control patients matched on day of admission | Total n = 580 | NR | Among 26 patients with a cardiac arrest, 20 (76.9%, 95% CI 60.7–93.1%) did not have a preceding positive early warning sign. |
| Case–control study, USA | Age 61 | n = 262 (cases) | |||
| Urban academic medical center, 745 beds | 49.6% males | n = 318 (controls) | |||
| Smith et al., 2013 (11) | General medical patients | Patients without an event | Total n = 35,585 patient episodes | Death within 24 h | Cardiac arrest within 24 h |
| Prospective cohort, UK | Age 67.7 | AUROC: 0.89 | AUROC (95% CI): | ||
| Urban hospital | 47.5% males | 95% CI 0.89–0.90) | 0.86 (0.85–0.87) | ||
| Churpek et al., 2014 (14) | Hospital ward patients | Patients without events | Total n = 59,301 | NR | Cardiac arrest within 24 h |
| Age | |||||
| Retrospective cohort study, USA | 55 (controls) | Novel EWS AUROC (95% CI): 0.88 (0.88–0.89) | |||
| Academic tertiary care hospital, 500 beds | 64 (cardiac arrest) | ViEWS AUROC (95% CI): | |||
| Males | 0.74 (0.72–0.75) | ||||
| 43% (controls) | |||||
| 41% (cardiac arrest) |
In general, the discriminative performance of EWS scores for predicting death within 48 hours of measurement was high (8, 9, 11, 13). In the largest study (n = 75,419), the researchers used the VitalPAC Early Warning Score (ViEWS), an abbreviated version of a previously derived EWS with vital signs recorded using VitalPAC software (8). Among medical ward patients, the abbreviated ViEWS score had an area under the receiver operator characteristic curve (AUROC) of 0.89 (95% confidence interval [CI], 0.85–0.92). The AUROC values remained above 0.85 in subgroup analyses, including sex, year of admission, age, indication for admission, and specific diagnoses. Two other studies of the same cohort found EWS models performed similarly well for predicting death within 24 hours (ViEWS, 0.888 [95% CI, 0.880–0.895] and National Early Warning Score [NEWS], 0.894 [95% CI, 0.887–0.902]) (9, 11). In a study conducted in a resource-poor hospital in Uganda without an ICU, the researchers reported that the ViEWS had an AUROC of 0.886 (95% CI, 0.826–0.947) for death within 24 hours (13).
Despite good discriminative ability, these studies also found clinically important trade-offs in sensitivity and specificity when using specific scores. Kellett and Kim found that low abbreviated ViEWS scores were associated with a very good prognosis: 0.02% of 49,077 patients with scores <3 (of 21 possible points) died within 48 hours. Although high scores were associated with a poorer prognosis (14% of 519 patients with scores >11 died within 48 hours), most patients (86%) with high scores survived (8). Prytherch and coworkers determined that the sensitivity of the ViEWS was approximately 67% at a specificity of 90% (9).
Studies also found consistently good predictive performance of EWS scores for cardiac arrest (7, 11, 12, 14). The NEWS score had an AUROC of 0.857 (95% CI, 0.847–0.868) as a predictor of cardiac arrest within 24 hours (11). The Cardiac Arrest Risk Triage (CART) EWS, a four-item model, had relatively similar predictive ability for cardiac arrest within 48 hours (AUROC 0.84) and was superior to a more comprehensive Modified Early Warning Score (MEWS) model among patients admitted to medical and surgical units (MEWS, 0.78; P = 0.001) (12). Churpek and colleagues found similar results in a nested case–control study (88 cases of cardiac arrest, 352 controls) of MEWS using data from the same cohort (AUROC, 0.77; 95% CI, 0.71–0.82) (7). They also developed and cross-validated a novel 16-item EWS score using the electronic health records of all patients on the hospital wards over a 33-month period and found it to be superior to the ViEWS EWS score in predicting cardiac arrest within 24 hours (AUROC of 0.88 [95% CI, 0.88–0.89] vs. AUROC of 0.74 [95% CI, 0.72–0.75]) (14).
As with studies of mortality, these studies reported high discriminative performance for predicting cardiac arrest, but potentially important trade-offs in sensitivity and specificity. At similar triggering scores, a specificity of 90% was associated with a sensitivity of 53% for CART and 48% for MEWS (12). Lowering the MEWS triggering score from 4 to 3 (of a possible 14) decreased both the specificity (87%) and the sensitivity (51%) (12). At a sensitivity of 60%, the comprehensive model developed by Churpek and coworkers had a specificity of 95%, whereas the ViEWS model had a specificity of 85% (14).
Rothschild and colleagues conducted a case–control study (n = 580) of at least one early warning criterion within 8 hours of a life-threatening event among patients on the medical ward (10). Criteria most associated with a life-threatening event included respiratory rate >35 breaths/min (odds ratio [OR], 31.1; 95%, CI, 7.5–129.6), need for supplemental oxygen to 100% or use of a non-rebreathing mask (OR, 13.7; 95% CI, 5.4–35), and heart rate >140 beats/min (OR, 8; 95% CI, 2.4–27.5). Multiple positive criteria were more common in cases than in controls (three positive criteria; 8.4% vs. 4.9%, P = 0.00027). However, among the 26 patients with a cardiac arrest, 20 (77%) did not have a preceding positive early warning sign within 8 hours of the event.
Though the included studies on predictive ability were generally well conducted for their study design and provide important information, the body of evidence is limited by some risk of bias (Appendix D). The case–control design of two of the studies (7, 10) has the potential for differential exposure assessment, as it is unclear whether vital sign measurement was different for cases compared with controls. In addition, although three studies were designed for validation of models (7, 8, 13), five were derivation studies (9–12, 14) and were not validated in separate populations of patients. Derivation studies are generally at risk for overfitting data to the population under study and may not be broadly applicable to populations of interest.
We found one randomized controlled trial (15) and 12 pre–post observational studies (16–27) that reported on the impact of EWS implementation on outcomes of 30-day mortality, cardiac arrest, and utilization of resources (Table 3). We found no studies reporting on the use of vasopressors, number of ventilator days, or respiratory failure. The EWS models ranged in scoring criteria from 5 to 12 items; all included heart rate, respiratory rate, and systolic blood pressure; and most included level of consciousness or mental status, temperature, and urinary output (Table 1). The studies were conducted in several countries (six in the United Kingdom, three in the United States, three in Australia, and one in Belgium), and most were carried out in urban academic hospitals.
| Study, design, setting | Observation period | Population, mean age (yr), sex | Score that triggers activation of response, description of response | In-hospital outcomes: mortality (30-d), cardiovascular events | Resource use: LOS, ICU | Resource use: MET, RRT, critical care outreach teams, or nursing |
|---|---|---|---|---|---|---|
| Albert and Huesman 2011 (16) | 6 mo | Telemetry ward n = 140 | MEWS: 1 or 2; nurse alert to reassess patient in ≤ 4 h | NR | NR | RRT calls: |
| Retrospective cohort study, USA | Age | MEWS≥ 3 | n = 78 (55.7%) | |||
| Tertiary care academic hospital, 550 beds | 64.7 (RRT) | Discuss with charge nurse and decision made on RRT activation based on stability | Mean MEWS for RRT 6.35 | |||
| 65.8 (code) | 6 mo after implementation: | |||||
| 33% increase in code team calls and 50% increase in RRT calls | ||||||
| De Meester et al., 2013 (17) | 5-d period after ICU discharge | n = 1,039 | +2 MEWS change score of 3 on one criterion; nurse felt patient unsafe | Patients who died without a DNAR or readmission to ICU (%), pre vs. post: | NR | Nursing compliance 53% Patient observation frequency per nursing shift increased from 0.9993 (95% CI 0.9637–1.0035) to 1.07 (95% CI, 1.10362–1.1101), P = 0.005 |
| Pre–post study, Belgium | Age 59 | Increase frequency of patient observations and physician notified | 5.7 vs. 3.5, NS | Nighttime: no difference. | ||
| Tertiary care hospital, 14 medical and surgical wards | 60% males | Daytime: increase from 1.1404 (95% CI, 1.1067–1.1742) to 1.2262 (95% CI, 1.1899–1.2625), P = 0.001 | ||||
| Green and Williams, 2006 (18) | Pre: 1 yr | n = 415 | NR | Mortality (%), pre vs. post: | Hospital: | Pre vs. post: |
| Pre–post study, Australia | Demographics NR | Activate ICU liaison team | 33.9 vs. 34.5, NS | 19 d (pre) | Code blue calls for cardiac arrest: 52.1% vs. 35%, NS | |
| Tertiary care academic hospital surgical and medical wards, 323 beds | Post: 2 yr | 16 d (post), NS | Code blue calls in patients who were still breathing and with a pulse: | |||
| ICU: | 47.9% vs. 64.6%, P = 0.0003 | |||||
| n = 153 (pre) including 111 admissions and 42 readmissions | ICU liaison team visits: | |||||
| n = 412 (post) including 320 admissions and 92 readmits, NS | 630 patients (1,958 visits) vs. 1,889 patients (4,586 visits) | |||||
| LOS (median): | ICU admissions with clinical markers present | |||||
| Pre: 3.0 (1.3–6.9) | <6 h: 58.8% vs. 75.4% | |||||
| Post: 2.6 (1.2–6.4), NS | ≥6 h: 41.2% vs. 24.5%, P < 0.002 | |||||
| (P < 0.0002) | ||||||
| Hammond et al., 2013 (27) | Pre: 1 month | Pre: n = 69 | MEWS | NR | ICU readmissions (%): 13 vs. 26, NS | Nursing, full vs. frequency: |
| Pre–post study, Australia | Discharged from ICU: | LOS ICU: 3 vs 3, NS | Within 24 h of ICU discharge: 210% increase (P = NR) | |||
| Tertiary care hospital | Post: 1 month | Age 64 | LOS (median), d | Within 1 hr of ICU discharge: 32% increase, P = 0.002 | ||
| 58% males | 13 (9–19) vs | |||||
| 24 h after being discharged from ICU or being admitted to ICU | Unplanned admission to ICU: | 17 (10–27), NS | ||||
| Age 60, 36% males | ||||||
| Post: n = 70 | ||||||
| Discharged from ICU: | ||||||
| Age 64 | ||||||
| 67% males | ||||||
| Unplanned admission to ICU: | ||||||
| Age 69 | ||||||
| 53% males | ||||||
| Jones et al., 2011 (19) | Pre: 47 d | Pre: n = 705 (7,820 observations) | EWS 3–5 (total 14): inform charge nurse and nurse intervention, recheck in 1 hr, if still ≥ 3, call junior physician, if still ≥ 3 in 1 hr, call senior physician and recheck 1 hr, if still ≥ 3 call critical care medical team | Pre vs. post: | Pre vs. post: | Pre vs. post: |
| Pre–post study, UK | Age 70 (median) | Mortality (%): | Hospital (d): | Clinical attendance (%) with EWS 3–5: 29 vs. 78, P < 0.001 | ||
| Clinical attendance (%) with EWS > 5: 67 vs. 96.2, P = 0.003 | ||||||
| Academic hospital medical assessment unit | Post: 38 d | 52% males | 9.5 vs. 7.6, P = 0.19 | 9.7 vs. 6.9, P < 0.001 | Clinical attendance (%) for EWS 3, 4, or 5 (automated alerts): | |
| Post: n = 776 (5,848 observations) | Cardiac arrest (%): | ICU (d): | 29% vs. 78%, P < 0.001 | |||
| Age 65 (median) | 0.4 vs. 0, P = 0.21 | 51 vs. 26, P = 0.04 | Clinical attendance (%) for EWS > 5: 67 vs. 95, P < 0.001 | |||
| 53% males | EWS accuracy (%) with electronic calculation: 81–100 | |||||
| Maupin et al., 2009 (20) | Pre: 1 year | n = NR | EWS = 3: increased VS frequency | NR | NR | RRT calls per 1,000 patient days increased from 7.8 to 16.4 after implementation |
| Pre–post study, USA | EWS = 4: notify physician | |||||
| General hospital, 200 beds | Post: 1 year | EWS = 5: call RRT (total 14) | ||||
| General medical and surgical wards | ||||||
| Mitchell et al., 2010 (21) | Pre: 4 mo | Pre: | MEWS≥ 4 | NR | ICU admissions (%): 1.8 vs. 0.5 | NR |
| Pre–post study, Australia | n = 1,157 | ≥4: Contact intern | RRR = 0.28 | |||
| 2 teaching hospitals | Post: 4 mo | Age 58.6 | ≥6: Contact registrar and intern to review in 30 min | (95% CI, 0.11–0.74), P = 0.006 | ||
| 55.7% males | LOS: | |||||
| Post: | Pre: 4.0 (1.8–8.3) | |||||
| n = 985 | Post: 4.8 (2.2–9.8), P = 0.03 | |||||
| Age 57.4 | ≥8: Registrar and consultant notified, consider MET | |||||
| 54.8% males | ||||||
| Moon et al., 2011 (22) | Pre: 3 yr | Pre: | MEWS >1 (total 21) | Pre vs. post: | Pre vs. post: | Hospital deaths per cardiac arrest call (%), pre vs. post: |
| Pre–post study, UK | n = 213,117 (FH) | Graduated response for MEWS scores >1 | Mortality of patients undergoing CPR (%): | LOS: NR | FH: 26 vs. 21, P < 0.0001 | |
| Two academic tertiary care hospitals: | Post: 3 yr | n = 248,260 (RVI) | FH: 52 vs. 42, P = 0.05 | % ICU admissions after undergoing in-hospital CPR: | RVI: 19.5 vs. 18.5, P = 0.12 | |
| Freeman Hospital (FH) | Post: | RVI: 70 vs. 40, P < 0.0001 | FH: 3 vs. 2, P = 0.004 | |||
| Royal Victoria Infirmary (RVI) | n = 235,516 (FH) | Hospital deaths (%): | RVI: 7 vs. 3, P < 0.0001 | |||
| n = 281,831 (RVI) | FH: 1.4 vs. 1.2, P < 0.0001 | Total ICU admissions per year: | ||||
| Demographics NR | RVI: 1.5 vs. 1.3, P < 0.0001 | FH: 857 vs. 1135 | ||||
| Cardiac arrests (% of all admissions): | (24.5% increase) | |||||
| FH: 0.36 vs. 0.25, P < 0.0001 | RVI: 538 vs. 827 | |||||
| RVI: 0.29 vs. 0.24, P < 0.0001 | (54% increase) | |||||
| Odell et al., 2002 (23) | Pre: 7 mo | n = NR | MEWS = 3 (total 15) | NR | NR | Calls to outreach increased from 432 in 7 mo to 231 in 3 mo |
| Pre–post study, UK | Referral to patient’s medical team and the critical care outreach team | |||||
| Surgical wards | Post: 3 month pilot | |||||
| Patel et al., 2011 (24) | Pre: 3 yr | n = 32,149 (admissions) | MEWS >4 (total 21) | Decrease (%) in deaths per admissions, pre vs. post: | NR | NR |
| Pre–post study, UK | Age NR | Nursing to seek senior medical advice, referral to RRT (critical care outreach team) | Males: 0.4 (95% CI, 0.003–0.81), P = 0.214 | |||
| Trauma and orthopedic wards | Post: 3 yr | 55% males | Females 1.5% (95% CI, 0.81–2.21), P = 0.108 | |||
| Total: 0.9 (95% CI, 0.53–1.31%), P = 0.092 | ||||||
| Smith and Oakey, 2006 (25) | 21 d (during outbreak) | n = 89 | MEWS = 3 (total 17) | NR | NR | Median 4.9 observation sets/patient day |
| Retrospective cohort study with matched control, UK | Median age: | Referral for critical care advice | Median 3.6 EWS/patient day; 2,036/3,739 (54.4%) observation sets contained a correct EWS | |||
| General hospital | Cases 64.7 | Uo and LOC inconsistently recorded | ||||
| Ward patients with suspected Legionnaire's disease during an outbreak | Controls 61.0 | RR had highest scoring errors 264/2,757 errors (9.6%) vs. HR 5.4%, SBP 4.3% vs. Temp 3.9% | ||||
| 66/270 (24.4%) observation sets were underscored and should have triggered an intervention but did not proportion of incorrect scores higher in the cases (17%) vs. control (12%) for difference of 5% (95% CI, 0–10.7), P = 0.02 | ||||||
| Subbe et al., 2003 (26) | Pre: 2 mo | n = 1,695 (case) | MEWS >4 (total 15) | MEWS 0–2: | Hospital: NR | NR |
| Pre–post study, UK | n = 659 (control) | Doctors examined patients within 1 hr | 6% (case) | ICU: | ||
| General hospital medical admissions unit, 56 beds | Post: 2 mo | Age: | 6% (control), P = 0.97 | |||
| 64 (case) | MEWS 3–4: | LOS in ICU, | ||||
| 63 (control) | 17% (case) | 2 day (case) | ||||
| Males: | 13% (control) | 4 day (control), P = 0.3 | ||||
| 4% (case) | P = 0.29 | |||||
| 45% (control) | MEWS 5–15: | |||||
| 28% (case) | ||||||
| 20% (control) | ||||||
| P = 0.25 | ||||||
| Cardiopulmonary arrest, score of 3 or 4: | ||||||
| 5% (case) | ||||||
| 0% (control) P < 0.016 | ||||||
| Bailey et al., 2013 (15) | 12 mo | Trial group | Real-time automated alerts score: NR | Mortality: 11% in intervention group, 10% in control group, NS | ICU Txf: 16% in intervention group, 14% in control group, NS | NR |
| Randomized controlled crossover study, USA | n = 2,353 | Charge nurse assessed the patient and alerted covering physician | Length of stay, median: 7.07 vs. 6.92, NS | |||
| 1,250-bed tertiary care academic hospital, general medical ward | Age 57 | |||||
| 53% males | ||||||
| 50% white, 48% black |
Seven studies addressed the effects of EWS implementation on mortality and had mixed results (15, 17–19, 22, 24, 26). In one good-quality trial (15) and five pre–post observational studies (18, 19, 22, 24, 26), no significant differences in overall mortality were observed. Only one study found a statistically significant decrease in mortality after EWS implementation (22). Moon and colleagues studied two hospitals and found that deaths per adult admission decreased from 1.4 to 1.2% (P < 0.0001) in one hospital and from 1.5 to 1.3% (P < 0.0001) in the other (22). They also found that patients who had undergone cardiopulmonary resuscitation had a significant decrease in in-hospital mortality at the two hospitals, from 52 to 42% (P < 0.05) and from 70 to 40% (P < 0.0001), respectively (22). Notably, the hospitals underwent an expansion of critical care outreach services during this period, so the independent impact of EWS implementation is unknown.
Mixed results were also found in the four studies evaluating the effects of EWS implementation on cardiac arrests (18, 19, 22, 26). Moon and colleagues studied the proportion of cardiac arrest calls per adult admission and found a decrease in the two hospitals involved in the study, from 0.4 to 0.2% at one site (P < 0.0001) and from 0.34 to 0.28% (P < 0.0001) at the other (22). Green and Williams found a significant decrease in the proportion of patients who had had a cardiac arrest (lack of pulse or respirations) at the time of the “code blue” call, from 52.1 to 35% (P = 0.0024) (18). Subbe and coworkers stratified the incidence of cardiac arrest by EWS score at time of admission and found no difference between low-risk (scores 0–2) and high-risk (scores 5–15) groups after EWS implementation, whereas the moderate-risk group (scores 3 or 4) demonstrated a significant increase in cardiac arrest (5% vs. 0%, P < 0.016) (26). No difference was found in the incidence of cardiac arrest after EWS implementation in the study by Jones and coworkers (19).
Four studies evaluated length of hospital stay before and after EWS implementation, with inconsistent results (15, 18, 19, 21). In a good-quality trial (15) and a pre–post observational study (18), no differences in length of hospital stay were detected 1–2 years after EWS implementation. A study with a shorter observation period (47 days before and 38 days after EWS implementation) found a significantly reduced length of stay (median [interquartile range (IQR)] 9.7 days [4.70–19.8] vs. 6.9 days [3.3–13.9]; P < 0.001) (19). An increase in length of hospital stay from 4.0 (1.8–8.3) days to 4.8 (2.2–9.8) days was observed in a fourth study comparing data 4 months before with those 4 months after EWS implementation (21). Variation in study populations (e.g., all patients on a ward vs. selected patients) and follow-up time make it difficult to assess the overall effect of EWS on length of stay across studies.
Five studies evaluating the impact of the EWS on ICU utilization also reported mixed results (15, 18, 19, 22, 26). Two studies found a significant increase in the number of ICU admissions after implementing EWS and accounting for differences in overall hospital admission rates (18, 19), whereas a third study found no difference in the proportion of patients transferred from the general medicine wards to the ICUs (15). One study involving two hospitals found increases of 24.5% and 14% in the annual ICU admission rates, but a significant decrease in the proportion of patients admitted to the ICU after having undergone cardiopulmonary resuscitation (pre-EWS 3% vs. post-EWS 2%, P = 0.004; and pre-EWS 6.65% vs. post-EWS 2.63%, P < 0.001) (22). One study found the proportion of clinically unstable patients who were on the ward for 6 hours or longer had decreased from 41.2 to 24.5% after implementing EWS (18). Two studies found no difference in length of ICU stay (18, 26).
Four studies evaluated the impact of EWS on RRTs and code teams (16, 18, 20, 22). All of the studies found at least a 50% increase in the number of RRT or ICU liaison team calls. Code blue calls decreased by 6–33% in three studies (16, 18, 22). One study found that the number of code blue calls for a patient still breathing and with a pulse increased from 47.9 to 64.4%, suggesting that response teams were activated before the patient’s condition deteriorated to the point of cardiac or respiratory arrest (18).
Four studies examined the impact of EWS implementation on nursing (17, 19, 25, 27). Three studies measured the accuracy and compliance of scoring (17, 19, 25). Compliance was as low as 53% in one study (17) and as high as 81–100% with the use of electronic calculations (19). Smith and Oakey found that the most inconsistently recorded data were urinary output and level of consciousness, with 45.6% missing values, and that respiratory rate had the highest errors (9.6%) (25). Hammond and coworkers found that urinary output was the only statistically significant improvement in individual vital sign recordings after EWS implementation (27% increase, P = 0.03) (27). Patient observations and clinical attention by nursing staff increased with EWS implementation, with greater attention given to patients with higher scores (19). One study found that the frequency of patient observations increased during daytime shifts, but not at night (17). Two studies examined the period after discharge from the ICU, with one study finding a significant increase in full vital sign documentation during the first 24 hours post-ICU discharge (P < 0.001) (27) and the other finding an increase in the frequency of patient observations per nursing shift (17).
Although the one randomized controlled study was of good quality, all of the other included studies used an historically controlled study design, which has an inherently high risk of bias because of the presence of unmeasured confounding factors. Advances in medical care or other changes in practice cannot be ruled out. None of the included retrospective studies adjusted for preintervention trends in mortality rate or accounted for other secular changes in care that could simultaneously have affected mortality, and none followed changes in outcome rates over time following implementation. Immortal time bias is another potential cofounder, in that patients without adverse events have more time available for vital sign measurement. The impact of EWS implementation on health outcomes and resource utilization is inconclusive because of these methodological limitations in the evidence.
EWS tools, mostly using vital sign abnormalities, perform reasonably well in predicting cardiac arrest and death within 48 hours of measurement. The effects of EWS use on health outcomes and utilization of resources, however, remain uncertain.
A priori, for predictive ability, we elected to include only studies that reported on critical events within 48 hours after the scores were intended to identify patients with more immediate critical needs so that interventions could be implemented before the actual event occurred. Death occurring beyond the acute time frame may actually reflect a subsequent change rather than the score that triggered the initial intervention. Kellett and Kim, in addition to reporting death within 48 hours, also reported on death at multiple time points up to 30 days and found good predictability for death over time (AUROCs: 0.93 at 48 hours, 0.87 at 5 days, and 0.81 at 30 days) (8). The ability of an EWS to predict both short- and long-term death may be attributed to the fact that patients with profound vital sign abnormalities are at high risk of death, regardless of specific interventions or their timeliness. Indeed, Opio and colleagues, in a study conducted in a resource-poor setting in Uganda without an ICU, found an AUROC for mortality similar to other EWS scores, suggesting that patients with severe vital sign abnormalities are at high risk of dying, regardless of their level of care (13). In this article, we report on the predictability of clinically meaningful events (cardiopulmonary arrest, mortality), rather than on the use of RRT or admission to ICU, because these are the intended consequences of elevated scores and may be associated with unintended negative effects. The impact of RRT implementation was recently studied in a large, historically controlled study in a U.S. hospital with an around-the-clock ICU consultation service and found an increased risk of hospital death in patients transferred to the ICU from the ward (OR, 1.23; 95% CI, 1.09–1.49) (28). They found that RRT implementation did not improve the outcomes adjusted for severity of illness in patients transferred from the ward, and they expressed caution that RRT implementation may have unforeseen costs without obvious benefit (28).
Given the limitations of the design of pre–post studies, the impact of EWS implementation on in-hospital health outcomes and resource utilization has not been studied adequately to draw conclusions. With regard to scoring consistency, though RRT appeared to increase it and the use of programmed score calculators showed promise, all other outcomes were measured heterogeneously and had mixed results. Of note, the one randomized control study did not find any difference in mortality, transfers to ICU, or length of hospital stay (15). Additional studies of high quality are needed to determine if these findings are accurate in other populations.
The current evidence is limited by the methodological flaws of the pre–post study design: the widely varied characteristics in training and implementation, the lack of standardization of the score to trigger a response, and the heterogeneity of the response itself. In addition, the trade-offs in sensitivity and specificity for predicting mortality or cardiac arrest at specific cut-points make it a challenge to directly inform clinical decision-making. This leaves the applicability of our findings to real-world clinical settings uncertain, but does provide a good foundation on which to build future studies. Previous systematic reviews identified similar shortcomings in the existing evidence (1, 29, 30).
The most important limitation of our review is that, due to the paucity of available evidence, we included studies of low methodological quality to evaluate the impact of EWS implementation. Future research should include randomized trials as well as methodologically rigorous observational studies that include an active control arm with standardization of comparators. Clinically important health outcomes should include 30-day mortality, cardiopulmonary events, and quality of life at discharge when evaluating the impact of EWS implementation. Standardization of threshold scores and responses would help to improve the applicability of study findings. The use of electronic health records and software to calculate scores and alert staff may help to reduce heterogeneity. A recent study looking at vital sign changes within 24 hours of a cardiac arrest, ICU transfer, or death suggested that future studies consistently report on these three outcomes to allow for model comparison (31). We argue that ICU transfer is expected when a patient becomes unstable, which makes this a less meaningful outcome; however, we fully agree with the need for measuring only immediate critical outcomes (24- to 48-hour cardiac or pulmonary arrest and mortality).
EWSs are being used broadly without clear evidence of benefit in terms of patient outcomes. Recently, a working group of the American Thoracic Society recommended against implementing performance measures unless they demonstrate “clear benefit to patients and [are] supported by strong evidence” (32). The existing evidence on the impact of EWSs on patient outcomes is insufficient, and, without rigorously designed studies reporting clinically meaningful outcomes, there remains a risk of unintended consequences for resource utilization and cost.
Current early warning system tools perform well for predicting death and cardiac arrest within 48 hours, although the impact on in-hospital health outcomes and utilization of resources remains uncertain, owing to methodological limitations. Efforts to more accurately assess their effectiveness will be needed as use becomes more widespread.
| 1 . | McGaughey J, Alderdice F, Fowler R, Kapila A, Mayhew A, Moutray M. Outreach and Early Warning Systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev 2007;(3):CD005529. |
| 2 . | Whittington J, White R, Haig KM, Slock M. Using an automated risk assessment report to identify patients at risk for clinical deterioration. Jt Comm J Qual Patient Saf 2007;33:569–574. |
| 3 . | Berwick DM, Calkins DR, McCannon CJ, Hackbarth AD. The 100,000 lives campaign: setting a goal and a deadline for improving health care quality. JAMA 2006;295:324–327. |
| 4 . | Treadwell JR, Reston JT, Singh S, Talati R, McPheeters M. A Framework for “Best Evidence” Approaches in Systematic Reviews. Methods Research Report. (Prepared by the ECRI Institute Evidence-based Practice Center under Contract No. HHSA 290-2007-10063-I.) AHRQ Publication No. 11-EHC046-EF. Rockville, MD: Agency for Healthcare Research and Quality; June 2011. Available at: http://www.ncbi.nlm.nih.gov/books/NBK56653/pdf/TOC.pdf. |
| 5 . | Hayden JA, van der Windt DA, Cartwright JL, Côté P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med 2013;158:280–286. |
| 6 . | Agency for Healthcare Research Quality, Effective Health Care Program. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication No. 10(14)-EHC063-EF. Rockville, MD: Agency for Healthcare Research and Quality; January 2014 [accessed 2014 Jun 23]. Available from: http://www.effectivehealthcare.ahrq.gov/ehc/products/60/318/CER-Methods-Guide-140109.pdf |
| 7 . | Churpek MM, Yuen TC, Huber MT, Park SY, Hall JB, Edelson DP. Predicting cardiac arrest on the wards: a nested case-control study. Chest 2012;141:1170–1176. |
| 8 . | Kellett J, Kim A. Validation of an abbreviated VitalPAC™ Early Warning Score (ViEWS) in 75,419 consecutive admissions to a Canadian regional hospital. Resuscitation 2012;83:297–302. |
| 9 . | Prytherch DR, Smith GB, Schmidt PE, Featherstone PI. ViEWS—towards a national early warning score for detecting adult inpatient deterioration. Resuscitation 2010;81:932–937. |
| 10 . | Rothschild JM, Gandara E, Woolf S, Williams DH, Bates DW. Single-parameter early warning criteria to predict life-threatening adverse events. J Patient Saf 2010;6:97–101. |
| 11 . | Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation 2013;84:465–470. |
| 12 . | Churpek MM, Yuen TC, Park SY, Meltzer DO, Hall JB, Edelson DP. Derivation of a cardiac arrest prediction model using ward vital signs. Crit Care Med 2012;40:2102–2108. |
| 13 . | Opio MO, Nansubuga G, Kellett J. Validation of the VitalPAC™ Early Warning Score (ViEWS) in acutely ill medical patients attending a resource-poor hospital in sub-Saharan Africa. Resuscitation 2013;84:743–746. |
| 14 . | Churpek MM, Yuen TC, Park SY, Gibbons R, Edelson DP. Using electronic health record data to develop and validate a prediction model for adverse outcomes in the wards*. Crit Care Med 2014;42:841–848. |
| 15 . | Bailey TC, Chen Y, Mao Y, Lu C, Hackmann G, Micek ST, Heard KM, Faulkner KM, Kollef MH. A trial of a real-time alert for clinical deterioration in patients hospitalized on general medical wards. J Hosp Med 2013;8:236–242. |
| 16 . | Albert BL, Huesman L. Development of a modified early warning score using the electronic medical record. Dimens Crit Care Nurs 2011;30:283–292. |
| 17 . | De Meester K, Das T, Hellemans K, Verbrugghe W, Jorens PG, Verpooten GA, Van Bogaert P. Impact of a standardized nurse observation protocol including MEWS after Intensive Care Unit discharge. Resuscitation 2013;84:184–188. |
| 18 . | Green AL, Williams A. An evaluation of an early warning clinical marker referral tool. Intensive Crit Care Nurs 2006;22:274–282. |
| 19 . | Jones S, Mullally M, Ingleby S, Buist M, Bailey M, Eddleston JM. Bedside electronic capture of clinical observations and automated clinical alerts to improve compliance with an Early Warning Score protocol. Crit Care Resusc 2011;13:83–88. |
| 20 . | Maupin JM, Roth DJ, Krapes JM. Use of the Modified Early Warning Score decreases code blue events. Jt Comm J Qual Patient Saf 2009;35:598–603. |
| 21 . | Mitchell IA, McKay H, Van Leuvan C, Berry R, McCutcheon C, Avard B, Slater N, Neeman T, Lamberth P. A prospective controlled trial of the effect of a multi-faceted intervention on early recognition and intervention in deteriorating hospital patients. Resuscitation 2010;81:658–666. |
| 22 . | Moon A, Cosgrove JF, Lea D, Fairs A, Cressey DM. An eight year audit before and after the introduction of modified early warning score (MEWS) charts, of patients admitted to a tertiary referral intensive care unit after CPR. Resuscitation 2011;82:150–154. |
| 23 . | Odell M, Forster A, Rudman K, Bass F. The critical care outreach service and the early warning system on surgical wards. Nurs Crit Care 2002;7:132–135. |
| 24 . | Patel MS, Jones MA, Jiggins M, Williams SC. Does the use of a “track and trigger” warning system reduce mortality in trauma patients? Injury 2011;42:1455–1459. |
| 25 . | Smith AF, Oakey RJ. Incidence and significance of errors in a patient ‘track and trigger’ system during an epidemic of Legionnaires’ disease: retrospective casenote analysis. Anaesthesia 2006;61:222–228. |
| 26 . | Subbe CP, Davies RG, Williams E, Rutherford P, Gemmell L. Effect of introducing the Modified Early Warning score on clinical outcomes, cardio-pulmonary arrests and intensive care utilisation in acute medical admissions. Anaesthesia 2003;58:797–802. |
| 27 . | Hammond NE, Spooner AJ, Barnett AG, Corley A, Brown P, Fraser JF. The effect of implementing a modified early warning scoring (MEWS) system on the adequacy of vital sign documentation. Aust Crit Care 2013;26:18–22. |
| 28 . | Karpman C, Keegan MT, Jensen JB, Bauer PR, Brown DR, Afessa B. The impact of rapid response team on outcome of patients transferred from the ward to the ICU: a single-center study. Crit Care Med 2013;41:2284–2291. |
| 29 . | McNeill G, Bryden D. Do either early warning systems or emergency response teams improve hospital patient survival? A systematic review. Resuscitation 2013;84:1652–1667. |
| 30 . | Gao H, McDonnell A, Harrison DA, Moore T, Adam S, Daly K, Esmonde L, Goldhill DR, Parry GJ, Rashidian A, et al. Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med 2007;33:667–679. |
| 31 . | Churpek MM, Yuen TC, Edelson DP. Predicting clinical deterioration in the hospital: the impact of outcome selection. Resuscitation 2013;84:564–568. |
| 32 . | Kahn JM, Gould MK, Krishnan JA, Wilson KC, Au DH, Cooke CR, Douglas IS, Feemster LC, Mularski RA, Slatore CG, et al.; ATS Ad Hoc Committee on the Development of Performance Measures from ATS Guidelines. An official American Thoracic Society workshop report: developing performance measures from clinical practice guidelines. Ann Am Thorac Soc 2014;11:S186–S195. |
Supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Quality Enhancement Research Initiative project ESP 05-225, and the Portland VA Medical Center. C.G.S. is a Core Investigator in Health Services Research and Development at the Portland VA Medical Center and was supported by a VA HSR&D Career Development Award. The Department of Veterans Affairs did not have a role in the design and conduct of the study; the collection, management, analysis, and interpretation of the data; the preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or the U.S. government.
Author Contributions: M.E.B.S., J.C.C., M.O., D.K., A.R.Q., M.F., M.L.M., C.G.S.: contributed to the review of articles for inclusion, abstraction of data, and interpretation of the results. M.E.B.S., J.C.C., M.O., D.K., M.F., M.L.M., C.G.S.: contributed to preparation of the manuscript.
The article has an online supplement, which is accessible from this issue’s table of contents online at www.ats.journals.org.
Originally Published in Press as DOI: 10.1513/AnnalsATS.201403-102OC on October 8, 2014
Author disclosures are available with the text of this article at www.atsjournals.org.
