Annals of the American Thoracic Society

Rationale: Early warning system (EWS) scores are used by hospital care teams to recognize early signs of clinical deterioration and trigger more intensive care.

Objective: To systematically review the evidence on the ability of early warning system scores to predict a patient’s risk of clinical deterioration and the impact of early warning system implementation on health outcomes and resource utilization.

Methods: We searched the MEDLINE, CINAHL, and Cochrane Central Register of Controlled Trials databases through May 2014. We included English-language studies of early warning system scores used with adults admitted to medical or surgical wards. We abstracted study characteristics, including population, setting, sample size, duration, and criteria used for early warning system scoring. For predictive ability, the primary outcomes were modeled for discrimination on 48-hour mortality, cardiac arrest, or pulmonary arrest. Outcomes for the impact of early warning system implementation included 30-day mortality, cardiovascular events, use of vasopressors, respiratory failure, days on ventilator, and resource utilization. We assessed study quality using a modified Quality in Prognosis Studies assessment tool where applicable.

Measurements and Main Results: Of 11,183 citations studies reviewed, one controlled trial and 20 observational studies of 13 unique models met our inclusion criteria. In eight studies, researchers addressed the predictive ability of early warning system tools and found a strong predictive value for death (area under the receiver operating characteristic curve [AUROC], 0.88–0.93) and cardiac arrest (AUROC, 0.74–0.86) within 48 hours. In 13 studies (one controlled trial and 12 pre–post observational studies), researchers addressed the impact on health outcomes and resource utilization and had mixed results. The one controlled trial was of good quality, and the researchers found no difference in mortality, transfers to the ICU, or length of hospital stay. The pre–post designs of the remaining studies have significant methodological limitations, resulting in insufficient evidence to draw conclusions.

Conclusions: Early warning system scores perform well for prediction of cardiac arrest and death within 48 hours, although the impact on health outcomes and resource utilization remains uncertain, owing to methodological limitations. Efforts to assess performance and effectiveness more rigorously will be needed as early warning system use becomes more widespread.

Observational studies suggest that patients often show signs of clinical deterioration up to 24 hours prior to a serious clinical event requiring intensive interventions (1). Early warning system (EWS) scores are used by hospital care teams to recognize early signs of clinical deterioration and trigger more intensive care, such as increased nursing attention, informing the care provider, or activating a rapid response team (RRT) or medical emergency team (2). The use of EWSs has increased since the Institute for Healthcare Improvement launched its 100,000 Lives Campaign in 2004, in which the use of RRTs was recommended to save lives (3). Many hospital systems have started to implement the use of EWS scores, and, although evidence shows that this leads to an increase in the use of RRTs and admissions to the intensive care unit (ICU), uncertainty remains as to their ability to predict an impending health event and whether they are useful in reducing cardiac arrest and death (1). In this study, we assessed the ability of EWS tools to predict cardiac arrest, pulmonary arrest, or death outcomes within 48 hours of data collection, as well as to determine the impact of EWS use on in-hospital health outcomes and utilization of resources.

We developed an analytic framework (Appendix A) with input from key informants (clinicians, nurses, hospital administration members, and patient advocates) to visually display the key questions and scope parameters of the review. We searched the MEDLINE, CINAHL, and the Cochrane Central Register of Controlled Trials databases from their inception to May 2014 for English-language studies of EWS in adult medical and surgical ward populations (Appendix B). We obtained additional articles from systematic reviews and reference lists and by consulting experts. All citations were imported into an electronic database (EndNote X4; Thomson Reuters, New York, NY).

Investigators reviewed titles and abstracts of citations identified through literature searches, and eligible full-text articles were independently assessed by two reviewers based on predefined eligibility criteria (Appendix C). Disagreements were resolved through group consensus.

To assess the predictive ability of EWSs, we examined observational studies in which researchers reported associations between EWS scores and 48-hour mortality, cardiac arrest, or pulmonary arrest. To assess the impact of EWS implementation, we included studies in which researchers reported on 30 day mortality, cardiovascular events (cardiac arrest, acute coronary syndrome, and cardiogenic shock), use of vasopressors, number of days on a ventilator, respiratory failure, and utilization of resources, including ICU admission, use of RRT, and length of hospital stay. We used a best evidence approach to guide inclusion of studies (4).

Data Abstraction, Quality Assessment, and Data Synthesis

Data on population characteristics, study setting, number of subjects, number of subjects lost to full analysis, name and elements of scoring system, comparator, harms, implementation characteristics (e.g., staff training, pilot phase), and funding were abstracted from each study by one investigator and confirmed by a second investigator. For studies addressing the question of predictive ability, we abstracted data on model discrimination for outcomes of mortality, cardiac arrest, and pulmonary arrest within 48 hours. For all other studies, we abstracted patient health outcomes and resource utilization.

For studies addressing predictive ability, we adapted criteria described in the Quality in Prognosis Studies assessment tool (5). In this article, we discuss the strengths and limitations of the studies and describe the overall potential for bias as applicable. For trials addressing health outcomes, we used criteria and methods defined in the methods guide developed by the Agency for Healthcare Research and Quality (6). We synthesized data qualitatively for the pre–post observational studies because objective criteria are lacking to quantitatively rate or measure the quality in these types of studies.

From among 11,183 titles and abstracts, 202 articles were selected for full-text review (Figure 1). Of these, we included 21 studies, eight providing primary data on predictive value of EWS scores and 13 pertaining to the impact of EWS interventions. We found 13 unique models ranging from four to 16 items with scores based on aggregate weighted systems (Table 1).

Table 1. Parameters used in studies of the predictive ability and effectiveness of EWS scores for clinical deterioration in medical and surgical inpatients

Study, countryParameters, name of scoring systemParameters used in the scoring system
Heart rateRespiratory rateSBPTemperatureDecreased urinary outputO2 satDifficulty breathingIncrease in supplemental O2Mental Status (LOC)ConcernOther, specify
Rothschild et al., 2010 (10), USASingle items, not combinedxxxxxxxxxDBP, seizures, uncontrolled bleeding, change in color
Churpek et al., 2012 (12), USA4-item CARTxxDBP, age
Maupin et al., 2009 (20), USA5-item MEWSxxxxx
Jones et al., 2011 (19), UKPatientrack EWSxxxxx
Subbe et al., 2003 (26), UK5-item MEWSxxxxx
Churpek et al., 2012 (7), USA5-item MEWSxxxxx
O’Dell et al., 2002 (23), UK5-item MEWSxxxxx
De Meester et al., 2013 (17), Belgium6-item MEWSxxxxxx
Smith and Oakey, 2006 (25), UK6-item EWSxxxxxx
Patel et al., 2011 (24), UK6-item MEWSxxxxxxIndwelling Foley catheter
Kellett and Kim, 2012 (8), Canada6-item VIEWSxxxxxx
Bailey et al., 2013 (15), USA7-item EWSxxxShock index, DBP, anticoagulation use
Hammond et al., 2013 (27), Australia7-item MEWSxxxxxxDBP
Mitchell et al., 2010 (21)7-item MEWSxxxxxxx
Moon et al., 2011 (22), UK7-item MEWSxxxxxxx
Green and Williams, 2006 (18), Australia7-item clinical marker toolxxxxxxx
Smith et al., 2013 (11), UK7-item NEWSxxxxxxx
Opio et al., 2013 (13), Uganda7-item VIEWSxxxxxxx
Prytherch et al., 2010 (9), UK7-item VIEWSxxxxxxx
Albert and Huesman 2011 (16), USA12-item MEWSxxxxxxxxxxWBC count, new focal weakness
Churpek et al., 2014 (14), USA16-item EWSxxxxxxTime, prior ICU stay, age, DBP, BUN, potassium, anion gap, WBC, Hb, platelet count

Definition of abbreviations: BUN = blood urea nitrogen; CART = Cardiac Arrest Risk Triage; DBP = diastolic blood pressure; EWS = early warning system; Hb = hemoglobin; ICU = intensive care unit; LOC = level of consciousness; MEWS = Modified Early Warning Score; O2 Sat = oxygen saturation; SBP = systolic blood pressure; ViEWS = VitalPAC Early Warning Score; WBC = white blood cell.

Predictive Value of Early Warning Systems

Eight observational studies (six prospective cohort and two case–control) met our inclusion criteria (Table 2) (714). One study considered single predictors, and seven studies reported the predictive values of six distinct models of EWS scores for the outcomes of interest (death and cardiac arrest within 48 hours of measurement). No study reported on the predictive ability of EWS for respiratory arrest. Most of the models ranged from four to seven items, all of which included heart rate, respiratory rate, and blood pressure, and most included temperature and mental status. The studies were conducted in several countries (four in the United States, two in the United Kingdom, one in Canada, and one in Uganda); most were carried out in urban academic hospitals.

Table 2. Observational studies of the predictive value of EWS scores

Study, design, settingPopulation, mean age (yr), sexComparisonNumber of patientsPredictive measures of mortality occurring within 48 h of EWS data collection/analysisPredictive measures of cardiac arrest occurring within 48 h of EWS data collection/analysis
Churpek et al., 2012 (12)Medical/surgical patients, including telemetry patientsPatients without a cardiac arrest or ICU TxfTotal n = 47,427NRAUROC: 0.84 (95% CI, NR) vs. 0.78 for MEWS
Retrospective cohort study USAAgen = 88 (cardiac arrest)
Academic tertiary care hospital, 500 beds64 (cardiac arrest)n = 2,820 (ICU Txf)
60 (ICU Txf)n = 44,519 (control)
54 (control)
43% (cardiac arrest)
52% (ICU Txf)
43% (control)
Churpek et al., 2012 (7)Medical/surgical patientsPatients without cardiac arrestTotal n = 440NRMEWS AUC (95% CI)
Case–control study, USAAgen = 88 (cases)0.77 (0.71–0.82)
Academic tertiary care hospital, 500 beds64 (cases)n = 352 (controls)
58 (controls)
43% (cases)
49% (controls)
Kellett and Kim 2012 (8)Medical (non-ICU)/surgical inpatientsNoneTotal n = 75,41948-h mortality AUROC (95% CI)NR
Retrospective cohort, CanadaAge 63n = 43,693 (medical)All: 0.93 (0.91–0.95)
Urban academic hospital, 375 beds48.9% malen = 30,485 (surgical)Surgical: 0.89 (0.78–1.0)
Medical: 0.89 (0.85–0.92)
Opio et al., 2013 (13)Medical patientsPatients without an eventTotal n = 844Death within 24 hNR
Retrospective cohort, UgandaAge 45.2AUROC (95% CI):
Hospital, no ICU care, 330 beds42% males0.89 (0.82–0.95)
Prytherch et al., 2010 (9)General medicine and emergency patients (consecutively admitted)Patients alive at 24 h following observationTotal n = 35,585 patient episodesDeath within 24 hNR
Prospective cohort study UKAge 67.7AUROC (95% CI):
Urban hospital47.5% males0.89 (0.88–0.89)
Rothschild et al., 2010 (10)Medical and medicine subspecialty inpatientsControl patients matched on day of admissionTotal n = 580NRAmong 26 patients with a cardiac arrest, 20 (76.9%, 95% CI 60.7–93.1%) did not have a preceding positive early warning sign.
Case–control study, USAAge 61n = 262 (cases)
Urban academic medical center, 745 beds49.6% malesn = 318 (controls)
Smith et al., 2013 (11)General medical patientsPatients without an eventTotal n = 35,585 patient episodesDeath within 24 hCardiac arrest within 24 h
Prospective cohort, UKAge 67.7AUROC: 0.89AUROC (95% CI):
Urban hospital47.5% males95% CI 0.89–0.90)0.86 (0.85–0.87)
Churpek et al., 2014 (14)Hospital ward patientsPatients without eventsTotal n = 59,301NRCardiac arrest within 24 h
Retrospective cohort study, USA55 (controls)Novel EWS AUROC (95% CI): 0.88 (0.88–0.89)
Academic tertiary care hospital, 500 beds64 (cardiac arrest)ViEWS AUROC (95% CI):
Males0.74 (0.72–0.75)
43% (controls)
41% (cardiac arrest)

Definition of abbreviations: AUC = area under the curve; AUROC = area under the receiver operating characteristic curve; CI = confidence interval; EWS = early warning system; ICU = intensive care unit; ICU Txf = transfer to intensive care unit; MEWS = Modified Early Warning Score; NR = not reported; ViEWS = VitalPAC Early Warning Score.

Mortality Prediction.

In general, the discriminative performance of EWS scores for predicting death within 48 hours of measurement was high (8, 9, 11, 13). In the largest study (n = 75,419), the researchers used the VitalPAC Early Warning Score (ViEWS), an abbreviated version of a previously derived EWS with vital signs recorded using VitalPAC software (8). Among medical ward patients, the abbreviated ViEWS score had an area under the receiver operator characteristic curve (AUROC) of 0.89 (95% confidence interval [CI], 0.85–0.92). The AUROC values remained above 0.85 in subgroup analyses, including sex, year of admission, age, indication for admission, and specific diagnoses. Two other studies of the same cohort found EWS models performed similarly well for predicting death within 24 hours (ViEWS, 0.888 [95% CI, 0.880–0.895] and National Early Warning Score [NEWS], 0.894 [95% CI, 0.887–0.902]) (9, 11). In a study conducted in a resource-poor hospital in Uganda without an ICU, the researchers reported that the ViEWS had an AUROC of 0.886 (95% CI, 0.826–0.947) for death within 24 hours (13).

Despite good discriminative ability, these studies also found clinically important trade-offs in sensitivity and specificity when using specific scores. Kellett and Kim found that low abbreviated ViEWS scores were associated with a very good prognosis: 0.02% of 49,077 patients with scores <3 (of 21 possible points) died within 48 hours. Although high scores were associated with a poorer prognosis (14% of 519 patients with scores >11 died within 48 hours), most patients (86%) with high scores survived (8). Prytherch and coworkers determined that the sensitivity of the ViEWS was approximately 67% at a specificity of 90% (9).

Cardiac Arrest Prediction.

Studies also found consistently good predictive performance of EWS scores for cardiac arrest (7, 11, 12, 14). The NEWS score had an AUROC of 0.857 (95% CI, 0.847–0.868) as a predictor of cardiac arrest within 24 hours (11). The Cardiac Arrest Risk Triage (CART) EWS, a four-item model, had relatively similar predictive ability for cardiac arrest within 48 hours (AUROC 0.84) and was superior to a more comprehensive Modified Early Warning Score (MEWS) model among patients admitted to medical and surgical units (MEWS, 0.78; P = 0.001) (12). Churpek and colleagues found similar results in a nested case–control study (88 cases of cardiac arrest, 352 controls) of MEWS using data from the same cohort (AUROC, 0.77; 95% CI, 0.71–0.82) (7). They also developed and cross-validated a novel 16-item EWS score using the electronic health records of all patients on the hospital wards over a 33-month period and found it to be superior to the ViEWS EWS score in predicting cardiac arrest within 24 hours (AUROC of 0.88 [95% CI, 0.88–0.89] vs. AUROC of 0.74 [95% CI, 0.72–0.75]) (14).

As with studies of mortality, these studies reported high discriminative performance for predicting cardiac arrest, but potentially important trade-offs in sensitivity and specificity. At similar triggering scores, a specificity of 90% was associated with a sensitivity of 53% for CART and 48% for MEWS (12). Lowering the MEWS triggering score from 4 to 3 (of a possible 14) decreased both the specificity (87%) and the sensitivity (51%) (12). At a sensitivity of 60%, the comprehensive model developed by Churpek and coworkers had a specificity of 95%, whereas the ViEWS model had a specificity of 85% (14).

Single Vital Sign Prediction.

Rothschild and colleagues conducted a case–control study (n = 580) of at least one early warning criterion within 8 hours of a life-threatening event among patients on the medical ward (10). Criteria most associated with a life-threatening event included respiratory rate >35 breaths/min (odds ratio [OR], 31.1; 95%, CI, 7.5–129.6), need for supplemental oxygen to 100% or use of a non-rebreathing mask (OR, 13.7; 95% CI, 5.4–35), and heart rate >140 beats/min (OR, 8; 95% CI, 2.4–27.5). Multiple positive criteria were more common in cases than in controls (three positive criteria; 8.4% vs. 4.9%, P = 0.00027). However, among the 26 patients with a cardiac arrest, 20 (77%) did not have a preceding positive early warning sign within 8 hours of the event.

Quality Assessment.

Though the included studies on predictive ability were generally well conducted for their study design and provide important information, the body of evidence is limited by some risk of bias (Appendix D). The case–control design of two of the studies (7, 10) has the potential for differential exposure assessment, as it is unclear whether vital sign measurement was different for cases compared with controls. In addition, although three studies were designed for validation of models (7, 8, 13), five were derivation studies (912, 14) and were not validated in separate populations of patients. Derivation studies are generally at risk for overfitting data to the population under study and may not be broadly applicable to populations of interest.

Impact of Early Warning System Interventions

We found one randomized controlled trial (15) and 12 pre–post observational studies (1627) that reported on the impact of EWS implementation on outcomes of 30-day mortality, cardiac arrest, and utilization of resources (Table 3). We found no studies reporting on the use of vasopressors, number of ventilator days, or respiratory failure. The EWS models ranged in scoring criteria from 5 to 12 items; all included heart rate, respiratory rate, and systolic blood pressure; and most included level of consciousness or mental status, temperature, and urinary output (Table 1). The studies were conducted in several countries (six in the United Kingdom, three in the United States, three in Australia, and one in Belgium), and most were carried out in urban academic hospitals.

Table 3. Studies of the impact of EWS interventions on patient outcomes and resource utilization

Study, design, settingObservation periodPopulation, mean age (yr), sexScore that triggers activation of response, description of responseIn-hospital outcomes: mortality (30-d), cardiovascular eventsResource use: LOS, ICUResource use: MET, RRT, critical care outreach teams, or nursing
Albert and Huesman 2011 (16)6 moTelemetry ward n = 140MEWS: 1 or 2; nurse alert to reassess patient in ≤ 4 hNRNRRRT calls:
Retrospective cohort study, USAAgeMEWS≥ 3n = 78 (55.7%)
Tertiary care academic hospital, 550 beds64.7 (RRT)Discuss with charge nurse and decision made on RRT activation based on stabilityMean MEWS for RRT 6.35
65.8 (code)6 mo after implementation:
33% increase in code team calls and 50% increase in RRT calls
De Meester et al., 2013 (17)5-d period after ICU dischargen = 1,039+2 MEWS change score of 3 on one criterion; nurse felt patient unsafePatients who died without a DNAR or readmission to ICU (%), pre vs. post:NRNursing compliance 53% Patient observation frequency per nursing shift increased from 0.9993 (95% CI 0.9637–1.0035) to 1.07 (95% CI, 1.10362–1.1101), P = 0.005
Pre–post study, BelgiumAge 59Increase frequency of patient observations and physician notified5.7 vs. 3.5, NSNighttime: no difference.
Tertiary care hospital, 14 medical and surgical wards60% males  Daytime: increase from 1.1404 (95% CI, 1.1067–1.1742) to 1.2262 (95% CI, 1.1899–1.2625), P = 0.001
Green and Williams, 2006 (18)Pre: 1 yrn = 415NRMortality (%), pre vs. post:Hospital:Pre vs. post:
Pre–post study, Australia Demographics NRActivate ICU liaison team33.9 vs. 34.5, NS19 d (pre)Code blue calls for cardiac arrest: 52.1% vs. 35%, NS
Tertiary care academic hospital surgical and medical wards, 323 bedsPost: 2 yr16 d (post), NSCode blue calls in patients who were still breathing and with a pulse:
ICU:47.9% vs. 64.6%, P = 0.0003
n = 153 (pre) including 111 admissions and 42 readmissionsICU liaison team visits:
n = 412 (post) including 320 admissions and 92 readmits, NS630 patients (1,958 visits) vs. 1,889 patients (4,586 visits)
LOS (median):ICU admissions with clinical markers present
Pre: 3.0 (1.3–6.9)<6 h: 58.8% vs. 75.4%
Post: 2.6 (1.2–6.4), NS≥6 h: 41.2% vs. 24.5%, P < 0.002
(P < 0.0002)
Hammond et al., 2013 (27)Pre: 1 monthPre: n = 69MEWSNRICU readmissions (%): 13 vs. 26, NSNursing, full vs. frequency:
Pre–post study, Australia Discharged from ICU:LOS ICU: 3 vs 3, NSWithin 24 h of ICU discharge: 210% increase (P = NR)
Tertiary care hospitalPost: 1 monthAge 64LOS (median), dWithin 1 hr of ICU discharge: 32% increase, P = 0.002
 58% males13 (9–19) vs
24 h after being discharged from ICU or being admitted to ICUUnplanned admission to ICU:17 (10–27), NS
Age 60, 36% males
Post: n = 70
Discharged from ICU:
Age 64
67% males
Unplanned admission to ICU:
Age 69
53% males
Jones et al., 2011 (19)Pre: 47 dPre: n = 705 (7,820 observations)EWS 3–5 (total 14): inform charge nurse and nurse intervention, recheck in 1 hr, if still ≥ 3, call junior physician, if still ≥ 3 in 1 hr, call senior physician and recheck 1 hr, if still ≥ 3 call critical care medical teamPre vs. post:Pre vs. post:Pre vs. post:
Pre–post study, UK Age 70 (median)Mortality (%):Hospital (d):Clinical attendance (%) with EWS 3–5: 29 vs. 78, P < 0.001
Clinical attendance (%) with EWS > 5: 67 vs. 96.2, P = 0.003
Academic hospital medical assessment unitPost: 38 d52% males9.5 vs. 7.6, P = 0.199.7 vs. 6.9, P < 0.001Clinical attendance (%) for EWS 3, 4, or 5 (automated alerts):
 Post: n = 776 (5,848 observations)Cardiac arrest (%):ICU (d):29% vs. 78%, P < 0.001
Age 65 (median)0.4 vs. 0, P = 0.2151 vs. 26, P = 0.04Clinical attendance (%) for EWS > 5: 67 vs. 95, P < 0.001
53% malesEWS accuracy (%) with electronic calculation: 81–100
Maupin et al., 2009 (20)Pre: 1 yearn = NREWS = 3: increased VS frequencyNRNRRRT calls per 1,000 patient days increased from 7.8 to 16.4 after implementation
Pre–post study, USA EWS = 4: notify physician
General hospital, 200 bedsPost: 1 yearEWS = 5: call RRT (total 14)
General medical and surgical wards 
Mitchell et al., 2010 (21)Pre: 4 moPre:MEWS≥ 4NRICU admissions (%): 1.8 vs. 0.5NR
Pre–post study, Australia n = 1,157≥4: Contact internRRR = 0.28
2 teaching hospitalsPost: 4 moAge 58.6≥6: Contact registrar and intern to review in 30 min(95% CI, 0.11–0.74), P = 0.006
 55.7% malesLOS:
Post:Pre: 4.0 (1.8–8.3)
n = 985Post: 4.8 (2.2–9.8), P = 0.03
Age 57.4≥8: Registrar and consultant notified, consider MET
54.8% males
Moon et al., 2011 (22)Pre: 3 yrPre:MEWS >1 (total 21)Pre vs. post:Pre vs. post:Hospital deaths per cardiac arrest call (%), pre vs. post:
Pre–post study, UK n = 213,117 (FH)Graduated response for MEWS scores >1Mortality of patients undergoing CPR (%):LOS: NRFH: 26 vs. 21, P < 0.0001
Two academic tertiary care hospitals:Post: 3 yrn = 248,260 (RVI)FH: 52 vs. 42, P = 0.05% ICU admissions after undergoing in-hospital CPR:RVI: 19.5 vs. 18.5, P = 0.12
Freeman Hospital (FH) Post:RVI: 70 vs. 40, P < 0.0001FH: 3 vs. 2, P = 0.004
Royal Victoria Infirmary (RVI)n = 235,516 (FH)Hospital deaths (%):RVI: 7 vs. 3, P < 0.0001
n = 281,831 (RVI)FH: 1.4 vs. 1.2, P < 0.0001Total ICU admissions per year:
Demographics NRRVI: 1.5 vs. 1.3, P < 0.0001FH: 857 vs. 1135
Cardiac arrests (% of all admissions):(24.5% increase)
FH: 0.36 vs. 0.25, P < 0.0001RVI: 538 vs. 827
RVI: 0.29 vs. 0.24, P < 0.0001(54% increase)
Odell et al., 2002 (23)Pre: 7 mon = NRMEWS = 3 (total 15)NRNRCalls to outreach increased from 432 in 7 mo to 231 in 3 mo
Pre–post study, UK Referral to patient’s medical team and the critical care outreach team
Surgical wardsPost: 3 month pilot
Patel et al., 2011 (24)Pre: 3 yrn = 32,149 (admissions)MEWS >4 (total 21)Decrease (%) in deaths per admissions, pre vs. post:NRNR
Pre–post study, UK Age NRNursing to seek senior medical advice, referral to RRT (critical care outreach team)Males: 0.4 (95% CI, 0.003–0.81), P = 0.214
Trauma and orthopedic wardsPost: 3 yr55% malesFemales 1.5% (95% CI, 0.81–2.21), P = 0.108
 Total: 0.9 (95% CI, 0.53–1.31%), P = 0.092
Smith and Oakey, 2006 (25)21 d (during outbreak)n = 89MEWS = 3 (total 17)NRNRMedian 4.9 observation sets/patient day
Retrospective cohort study with matched control, UKMedian age:Referral for critical care adviceMedian 3.6 EWS/patient day; 2,036/3,739 (54.4%) observation sets contained a correct EWS
General hospitalCases 64.7Uo and LOC inconsistently recorded
Ward patients with suspected Legionnaire's disease during an outbreakControls 61.0RR had highest scoring errors 264/2,757 errors (9.6%) vs. HR 5.4%, SBP 4.3% vs. Temp 3.9%
66/270 (24.4%) observation sets were underscored and should have triggered an intervention but did not proportion of incorrect scores higher in the cases (17%) vs. control (12%) for difference of 5% (95% CI, 0–10.7), P = 0.02
Subbe et al., 2003 (26)Pre: 2 mon = 1,695 (case)MEWS >4 (total 15)MEWS 0–2:Hospital: NRNR
Pre–post study, UK n = 659 (control)Doctors examined patients within 1 hr6% (case)ICU:
General hospital medical admissions unit, 56 bedsPost: 2 moAge:6% (control), P = 0.97 
 64 (case)MEWS 3–4:LOS in ICU,
63 (control)17% (case)2 day (case)
Males:13% (control)4 day (control), P = 0.3
4% (case)P = 0.29
45% (control)MEWS 5–15:
28% (case)
20% (control)
P = 0.25
Cardiopulmonary arrest, score of 3 or 4:
5% (case)
0% (control) P < 0.016
Bailey et al., 2013 (15)12 moTrial groupReal-time automated alerts score: NRMortality: 11% in intervention group, 10% in control group, NSICU Txf: 16% in intervention group, 14% in control group, NSNR
Randomized controlled crossover study, USAn = 2,353Charge nurse assessed the patient and alerted covering physicianLength of stay, median: 7.07 vs. 6.92, NS
1,250-bed tertiary care academic hospital, general medical wardAge 57
53% males
50% white, 48% black

Definition of abbreviations: CI = confidence interval; CPR = cardiopulmonary resuscitation; DNAR = do not attempt resuscitation; EWS = Early Warning System; FH = Freeman Hospital; HR = heart rate; ICU = intensive care unit; ICU Txf = transfer to intensive care unit; LOC = level of consciousness; LOS = length of (hospital) stay; MET = medical emergency team; MEWS = Modified Early Warning Score; NR = not reported; NS = not significant; RR = respiratory rate; RRR = Relative Risk Ratio; RRT = Rapid Response Team; RVI = Royal Victoria Infirmary; SBP = systolic arterial pressure; Uo = urine output; VS = vital signs.


Seven studies addressed the effects of EWS implementation on mortality and had mixed results (15, 1719, 22, 24, 26). In one good-quality trial (15) and five pre–post observational studies (18, 19, 22, 24, 26), no significant differences in overall mortality were observed. Only one study found a statistically significant decrease in mortality after EWS implementation (22). Moon and colleagues studied two hospitals and found that deaths per adult admission decreased from 1.4 to 1.2% (P < 0.0001) in one hospital and from 1.5 to 1.3% (P < 0.0001) in the other (22). They also found that patients who had undergone cardiopulmonary resuscitation had a significant decrease in in-hospital mortality at the two hospitals, from 52 to 42% (P < 0.05) and from 70 to 40% (P < 0.0001), respectively (22). Notably, the hospitals underwent an expansion of critical care outreach services during this period, so the independent impact of EWS implementation is unknown.

Cardiac Arrest.

Mixed results were also found in the four studies evaluating the effects of EWS implementation on cardiac arrests (18, 19, 22, 26). Moon and colleagues studied the proportion of cardiac arrest calls per adult admission and found a decrease in the two hospitals involved in the study, from 0.4 to 0.2% at one site (P < 0.0001) and from 0.34 to 0.28% (P < 0.0001) at the other (22). Green and Williams found a significant decrease in the proportion of patients who had had a cardiac arrest (lack of pulse or respirations) at the time of the “code blue” call, from 52.1 to 35% (P = 0.0024) (18). Subbe and coworkers stratified the incidence of cardiac arrest by EWS score at time of admission and found no difference between low-risk (scores 0–2) and high-risk (scores 5–15) groups after EWS implementation, whereas the moderate-risk group (scores 3 or 4) demonstrated a significant increase in cardiac arrest (5% vs. 0%, P < 0.016) (26). No difference was found in the incidence of cardiac arrest after EWS implementation in the study by Jones and coworkers (19).

Length of Hospital Stay

Four studies evaluated length of hospital stay before and after EWS implementation, with inconsistent results (15, 18, 19, 21). In a good-quality trial (15) and a pre–post observational study (18), no differences in length of hospital stay were detected 1–2 years after EWS implementation. A study with a shorter observation period (47 days before and 38 days after EWS implementation) found a significantly reduced length of stay (median [interquartile range (IQR)] 9.7 days [4.70–19.8] vs. 6.9 days [3.3–13.9]; P < 0.001) (19). An increase in length of hospital stay from 4.0 (1.8–8.3) days to 4.8 (2.2–9.8) days was observed in a fourth study comparing data 4 months before with those 4 months after EWS implementation (21). Variation in study populations (e.g., all patients on a ward vs. selected patients) and follow-up time make it difficult to assess the overall effect of EWS on length of stay across studies.

Admissions to the ICU

Five studies evaluating the impact of the EWS on ICU utilization also reported mixed results (15, 18, 19, 22, 26). Two studies found a significant increase in the number of ICU admissions after implementing EWS and accounting for differences in overall hospital admission rates (18, 19), whereas a third study found no difference in the proportion of patients transferred from the general medicine wards to the ICUs (15). One study involving two hospitals found increases of 24.5% and 14% in the annual ICU admission rates, but a significant decrease in the proportion of patients admitted to the ICU after having undergone cardiopulmonary resuscitation (pre-EWS 3% vs. post-EWS 2%, P = 0.004; and pre-EWS 6.65% vs. post-EWS 2.63%, P < 0.001) (22). One study found the proportion of clinically unstable patients who were on the ward for 6 hours or longer had decreased from 41.2 to 24.5% after implementing EWS (18). Two studies found no difference in length of ICU stay (18, 26).

Use of Rapid Response and Code Teams

Four studies evaluated the impact of EWS on RRTs and code teams (16, 18, 20, 22). All of the studies found at least a 50% increase in the number of RRT or ICU liaison team calls. Code blue calls decreased by 6–33% in three studies (16, 18, 22). One study found that the number of code blue calls for a patient still breathing and with a pulse increased from 47.9 to 64.4%, suggesting that response teams were activated before the patient’s condition deteriorated to the point of cardiac or respiratory arrest (18).


Four studies examined the impact of EWS implementation on nursing (17, 19, 25, 27). Three studies measured the accuracy and compliance of scoring (17, 19, 25). Compliance was as low as 53% in one study (17) and as high as 81–100% with the use of electronic calculations (19). Smith and Oakey found that the most inconsistently recorded data were urinary output and level of consciousness, with 45.6% missing values, and that respiratory rate had the highest errors (9.6%) (25). Hammond and coworkers found that urinary output was the only statistically significant improvement in individual vital sign recordings after EWS implementation (27% increase, P = 0.03) (27). Patient observations and clinical attention by nursing staff increased with EWS implementation, with greater attention given to patients with higher scores (19). One study found that the frequency of patient observations increased during daytime shifts, but not at night (17). Two studies examined the period after discharge from the ICU, with one study finding a significant increase in full vital sign documentation during the first 24 hours post-ICU discharge (P < 0.001) (27) and the other finding an increase in the frequency of patient observations per nursing shift (17).

Quality Assessment

Although the one randomized controlled study was of good quality, all of the other included studies used an historically controlled study design, which has an inherently high risk of bias because of the presence of unmeasured confounding factors. Advances in medical care or other changes in practice cannot be ruled out. None of the included retrospective studies adjusted for preintervention trends in mortality rate or accounted for other secular changes in care that could simultaneously have affected mortality, and none followed changes in outcome rates over time following implementation. Immortal time bias is another potential cofounder, in that patients without adverse events have more time available for vital sign measurement. The impact of EWS implementation on health outcomes and resource utilization is inconclusive because of these methodological limitations in the evidence.

EWS tools, mostly using vital sign abnormalities, perform reasonably well in predicting cardiac arrest and death within 48 hours of measurement. The effects of EWS use on health outcomes and utilization of resources, however, remain uncertain.

A priori, for predictive ability, we elected to include only studies that reported on critical events within 48 hours after the scores were intended to identify patients with more immediate critical needs so that interventions could be implemented before the actual event occurred. Death occurring beyond the acute time frame may actually reflect a subsequent change rather than the score that triggered the initial intervention. Kellett and Kim, in addition to reporting death within 48 hours, also reported on death at multiple time points up to 30 days and found good predictability for death over time (AUROCs: 0.93 at 48 hours, 0.87 at 5 days, and 0.81 at 30 days) (8). The ability of an EWS to predict both short- and long-term death may be attributed to the fact that patients with profound vital sign abnormalities are at high risk of death, regardless of specific interventions or their timeliness. Indeed, Opio and colleagues, in a study conducted in a resource-poor setting in Uganda without an ICU, found an AUROC for mortality similar to other EWS scores, suggesting that patients with severe vital sign abnormalities are at high risk of dying, regardless of their level of care (13). In this article, we report on the predictability of clinically meaningful events (cardiopulmonary arrest, mortality), rather than on the use of RRT or admission to ICU, because these are the intended consequences of elevated scores and may be associated with unintended negative effects. The impact of RRT implementation was recently studied in a large, historically controlled study in a U.S. hospital with an around-the-clock ICU consultation service and found an increased risk of hospital death in patients transferred to the ICU from the ward (OR, 1.23; 95% CI, 1.09–1.49) (28). They found that RRT implementation did not improve the outcomes adjusted for severity of illness in patients transferred from the ward, and they expressed caution that RRT implementation may have unforeseen costs without obvious benefit (28).

Given the limitations of the design of pre–post studies, the impact of EWS implementation on in-hospital health outcomes and resource utilization has not been studied adequately to draw conclusions. With regard to scoring consistency, though RRT appeared to increase it and the use of programmed score calculators showed promise, all other outcomes were measured heterogeneously and had mixed results. Of note, the one randomized control study did not find any difference in mortality, transfers to ICU, or length of hospital stay (15). Additional studies of high quality are needed to determine if these findings are accurate in other populations.

The current evidence is limited by the methodological flaws of the pre–post study design: the widely varied characteristics in training and implementation, the lack of standardization of the score to trigger a response, and the heterogeneity of the response itself. In addition, the trade-offs in sensitivity and specificity for predicting mortality or cardiac arrest at specific cut-points make it a challenge to directly inform clinical decision-making. This leaves the applicability of our findings to real-world clinical settings uncertain, but does provide a good foundation on which to build future studies. Previous systematic reviews identified similar shortcomings in the existing evidence (1, 29, 30).

The most important limitation of our review is that, due to the paucity of available evidence, we included studies of low methodological quality to evaluate the impact of EWS implementation. Future research should include randomized trials as well as methodologically rigorous observational studies that include an active control arm with standardization of comparators. Clinically important health outcomes should include 30-day mortality, cardiopulmonary events, and quality of life at discharge when evaluating the impact of EWS implementation. Standardization of threshold scores and responses would help to improve the applicability of study findings. The use of electronic health records and software to calculate scores and alert staff may help to reduce heterogeneity. A recent study looking at vital sign changes within 24 hours of a cardiac arrest, ICU transfer, or death suggested that future studies consistently report on these three outcomes to allow for model comparison (31). We argue that ICU transfer is expected when a patient becomes unstable, which makes this a less meaningful outcome; however, we fully agree with the need for measuring only immediate critical outcomes (24- to 48-hour cardiac or pulmonary arrest and mortality).

EWSs are being used broadly without clear evidence of benefit in terms of patient outcomes. Recently, a working group of the American Thoracic Society recommended against implementing performance measures unless they demonstrate “clear benefit to patients and [are] supported by strong evidence” (32). The existing evidence on the impact of EWSs on patient outcomes is insufficient, and, without rigorously designed studies reporting clinically meaningful outcomes, there remains a risk of unintended consequences for resource utilization and cost.

Current early warning system tools perform well for predicting death and cardiac arrest within 48 hours, although the impact on in-hospital health outcomes and utilization of resources remains uncertain, owing to methodological limitations. Efforts to more accurately assess their effectiveness will be needed as use becomes more widespread.

The authors thank Rose Relevo, M.L.I.S., M.S., for developing the search strategy and compiling the literature database for this report.

1 . McGaughey J, Alderdice F, Fowler R, Kapila A, Mayhew A, Moutray M. Outreach and Early Warning Systems (EWS) for the prevention of intensive care admission and death of critically ill adult patients on general hospital wards. Cochrane Database Syst Rev 2007;(3):CD005529.
2 . Whittington J, White R, Haig KM, Slock M. Using an automated risk assessment report to identify patients at risk for clinical deterioration. Jt Comm J Qual Patient Saf 2007;33:569574.
3 . Berwick DM, Calkins DR, McCannon CJ, Hackbarth AD. The 100,000 lives campaign: setting a goal and a deadline for improving health care quality. JAMA 2006;295:324327.
4 . Treadwell JR, Reston JT, Singh S, Talati R, McPheeters M. A Framework for “Best Evidence” Approaches in Systematic Reviews. Methods Research Report. (Prepared by the ECRI Institute Evidence-based Practice Center under Contract No. HHSA 290-2007-10063-I.) AHRQ Publication No. 11-EHC046-EF. Rockville, MD: Agency for Healthcare Research and Quality; June 2011. Available at:
5 . Hayden JA, van der Windt DA, Cartwright JL, Côté P, Bombardier C. Assessing bias in studies of prognostic factors. Ann Intern Med 2013;158:280286.
6 . Agency for Healthcare Research Quality, Effective Health Care Program. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication No. 10(14)-EHC063-EF. Rockville, MD: Agency for Healthcare Research and Quality; January 2014 [accessed 2014 Jun 23]. Available from:
7 . Churpek MM, Yuen TC, Huber MT, Park SY, Hall JB, Edelson DP. Predicting cardiac arrest on the wards: a nested case-control study. Chest 2012;141:11701176.
8 . Kellett J, Kim A. Validation of an abbreviated VitalPAC™ Early Warning Score (ViEWS) in 75,419 consecutive admissions to a Canadian regional hospital. Resuscitation 2012;83:297302.
9 . Prytherch DR, Smith GB, Schmidt PE, Featherstone PI. ViEWS—towards a national early warning score for detecting adult inpatient deterioration. Resuscitation 2010;81:932937.
10 . Rothschild JM, Gandara E, Woolf S, Williams DH, Bates DW. Single-parameter early warning criteria to predict life-threatening adverse events. J Patient Saf 2010;6:97101.
11 . Smith GB, Prytherch DR, Meredith P, Schmidt PE, Featherstone PI. The ability of the National Early Warning Score (NEWS) to discriminate patients at risk of early cardiac arrest, unanticipated intensive care unit admission, and death. Resuscitation 2013;84:465470.
12 . Churpek MM, Yuen TC, Park SY, Meltzer DO, Hall JB, Edelson DP. Derivation of a cardiac arrest prediction model using ward vital signs. Crit Care Med 2012;40:21022108.
13 . Opio MO, Nansubuga G, Kellett J. Validation of the VitalPAC™ Early Warning Score (ViEWS) in acutely ill medical patients attending a resource-poor hospital in sub-Saharan Africa. Resuscitation 2013;84:743746.
14 . Churpek MM, Yuen TC, Park SY, Gibbons R, Edelson DP. Using electronic health record data to develop and validate a prediction model for adverse outcomes in the wards*. Crit Care Med 2014;42:841848.
15 . Bailey TC, Chen Y, Mao Y, Lu C, Hackmann G, Micek ST, Heard KM, Faulkner KM, Kollef MH. A trial of a real-time alert for clinical deterioration in patients hospitalized on general medical wards. J Hosp Med 2013;8:236242.
16 . Albert BL, Huesman L. Development of a modified early warning score using the electronic medical record. Dimens Crit Care Nurs 2011;30:283292.
17 . De Meester K, Das T, Hellemans K, Verbrugghe W, Jorens PG, Verpooten GA, Van Bogaert P. Impact of a standardized nurse observation protocol including MEWS after Intensive Care Unit discharge. Resuscitation 2013;84:184188.
18 . Green AL, Williams A. An evaluation of an early warning clinical marker referral tool. Intensive Crit Care Nurs 2006;22:274282.
19 . Jones S, Mullally M, Ingleby S, Buist M, Bailey M, Eddleston JM. Bedside electronic capture of clinical observations and automated clinical alerts to improve compliance with an Early Warning Score protocol. Crit Care Resusc 2011;13:8388.
20 . Maupin JM, Roth DJ, Krapes JM. Use of the Modified Early Warning Score decreases code blue events. Jt Comm J Qual Patient Saf 2009;35:598603.
21 . Mitchell IA, McKay H, Van Leuvan C, Berry R, McCutcheon C, Avard B, Slater N, Neeman T, Lamberth P. A prospective controlled trial of the effect of a multi-faceted intervention on early recognition and intervention in deteriorating hospital patients. Resuscitation 2010;81:658666.
22 . Moon A, Cosgrove JF, Lea D, Fairs A, Cressey DM. An eight year audit before and after the introduction of modified early warning score (MEWS) charts, of patients admitted to a tertiary referral intensive care unit after CPR. Resuscitation 2011;82:150154.
23 . Odell M, Forster A, Rudman K, Bass F. The critical care outreach service and the early warning system on surgical wards. Nurs Crit Care 2002;7:132135.
24 . Patel MS, Jones MA, Jiggins M, Williams SC. Does the use of a “track and trigger” warning system reduce mortality in trauma patients? Injury 2011;42:14551459.
25 . Smith AF, Oakey RJ. Incidence and significance of errors in a patient ‘track and trigger’ system during an epidemic of Legionnaires’ disease: retrospective casenote analysis. Anaesthesia 2006;61:222228.
26 . Subbe CP, Davies RG, Williams E, Rutherford P, Gemmell L. Effect of introducing the Modified Early Warning score on clinical outcomes, cardio-pulmonary arrests and intensive care utilisation in acute medical admissions. Anaesthesia 2003;58:797802.
27 . Hammond NE, Spooner AJ, Barnett AG, Corley A, Brown P, Fraser JF. The effect of implementing a modified early warning scoring (MEWS) system on the adequacy of vital sign documentation. Aust Crit Care 2013;26:1822.
28 . Karpman C, Keegan MT, Jensen JB, Bauer PR, Brown DR, Afessa B. The impact of rapid response team on outcome of patients transferred from the ward to the ICU: a single-center study. Crit Care Med 2013;41:22842291.
29 . McNeill G, Bryden D. Do either early warning systems or emergency response teams improve hospital patient survival? A systematic review. Resuscitation 2013;84:16521667.
30 . Gao H, McDonnell A, Harrison DA, Moore T, Adam S, Daly K, Esmonde L, Goldhill DR, Parry GJ, Rashidian A, et al. Systematic review and evaluation of physiological track and trigger warning systems for identifying at-risk patients on the ward. Intensive Care Med 2007;33:667679.
31 . Churpek MM, Yuen TC, Edelson DP. Predicting clinical deterioration in the hospital: the impact of outcome selection. Resuscitation 2013;84:564568.
32 . Kahn JM, Gould MK, Krishnan JA, Wilson KC, Au DH, Cooke CR, Douglas IS, Feemster LC, Mularski RA, Slatore CG, et al.; ATS Ad Hoc Committee on the Development of Performance Measures from ATS Guidelines. An official American Thoracic Society workshop report: developing performance measures from clinical practice guidelines. Ann Am Thorac Soc 2014;11:S186S195.
Correspondence and requests for reprints should be addressed to M. E. Beth Smith, D.O., Portland VA Medical Center, R&D 71, 3710 SW US Veterans Hospital Road, Portland, OR 97239. E-mail:

Supported by the Department of Veterans Affairs, Veterans Health Administration, Office of Research and Development, Quality Enhancement Research Initiative project ESP 05-225, and the Portland VA Medical Center. C.G.S. is a Core Investigator in Health Services Research and Development at the Portland VA Medical Center and was supported by a VA HSR&D Career Development Award. The Department of Veterans Affairs did not have a role in the design and conduct of the study; the collection, management, analysis, and interpretation of the data; the preparation, review, or approval of the manuscript; or the decision to submit the manuscript for publication. The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs or the U.S. government.

Author Contributions: M.E.B.S., J.C.C., M.O., D.K., A.R.Q., M.F., M.L.M., C.G.S.: contributed to the review of articles for inclusion, abstraction of data, and interpretation of the results. M.E.B.S., J.C.C., M.O., D.K., M.F., M.L.M., C.G.S.: contributed to preparation of the manuscript.

The article has an online supplement, which is accessible from this issue’s table of contents online at

Originally Published in Press as DOI: 10.1513/AnnalsATS.201403-102OC on October 8, 2014

Author disclosures are available with the text of this article at


No related items
Comments Post a Comment

New User Registration

Not Yet Registered?
Benefits of Registration Include:
 •  A Unique User Profile that will allow you to manage your current subscriptions (including online access)
 •  The ability to create favorites lists down to the article level
 •  The ability to customize email alerts to receive specific notifications about the topics you care most about and special offers
Annals of the American Thoracic Society

Click to see any corrections or updates and to confirm this is the authentic version of record