Comments
Abstract
Rationale: Accurate surveillance of ventilator-associated pneumonia (VAP) is hampered by subjective diagnostic criteria. A novel surveillance paradigm for ventilator-associated events (VAEs) was introduced.
Objectives: To determine the validity of surveillance using the new VAE algorithm.
Methods: Prospective cohort study in two Dutch academic medical centers (2011–2012). VAE surveillance was electronically implemented and included assessment of (infection-related) ventilator-associated conditions (VAC, IVAC) and VAP. Concordance with ongoing prospective VAP surveillance was assessed, along with clinical diagnoses underlying VAEs and associated mortality of all conditions. Consequences of minor differences in electronic VAE implementation were evaluated.
Measurements and Main Results: The study included 2,080 patients with 2,296 admissions. Incidences of VAC, IVAC, VAE-VAP, and VAP according to prospective surveillance were 10.0, 4.2, 3.2, and 8.0 per 1000 ventilation days, respectively. The VAE algorithm detected at most 32% of the patients with VAP identified by prospective surveillance. VAC signals were most often caused by volume overload and infections, but not necessarily VAP. Subdistribution hazards for mortality were 3.9 (95% confidence interval, 2.9–5.3) for VAC, 2.5 (1.5–4.1) for IVAC, 2.0 (1.1–3.6) for VAE-VAP, and 7.2 (5.1–10.3) for VAP identified by prospective surveillance. In sensitivity analyses, mortality estimates varied considerably after minor differences in electronic algorithm implementation.
Conclusions: Concordance between the novel VAE algorithm and VAP was poor. Incidence and associated mortality of VAE were susceptible to small differences in electronic implementation. More studies are needed to characterize the clinical entities underlying VAE and to ensure comparability of rates from different institutions.
Surveillance of ventilator-associated pneumonia is hampered by subjectivity. A novel surveillance paradigm for ventilator-associated events was introduced to address this concern.
A novel surveillance paradigm for ventilator-associated events was introduced to address this concern. The novel surveillance paradigm detects a broad range of clinical conditions; however, it has poor sensitivity for the detection of ventilator-associated pneumonia. As the paradigm appeared vulnerable to minor variations in electronic implementation, caution should be taken before considering the novel paradigm as an established quality metric.
Health care–associated infections add a considerable burden to medical care. Monitoring and preventing these infections have become increasingly important, along with a rise in (mandatory) public reporting (1–3). Surveillance networks such as the National Healthcare Safety Network (NHSN) allow for benchmarking among hospitals, and feedback of infection rates offers guidance for improvement programs (4, 5).
Ventilator-associated pneumonia (VAP) has been associated with increased mortality and length of stay. Therefore it is one of the major infections targeted by surveillance programs (6). However, establishing the diagnosis of VAP is challenging and concerns have been voiced regarding the reliability of VAP surveillance and its use as a tool for hospital benchmarking (7–9). VAP case definitions are complex, labor-intensive, and leave room for subjective interpretation (10). Low interrater reliability and poor correlation with histopathology have also been described (11, 12). In consequence, variations in implementation of VAP surveillance across hospitals affect the reported VAP rates and preclude valid interhospital comparisons (9, 11, 13). Furthermore, studies of health care–associated infection surveillance and prevention are vulnerable to assessment bias (14).
These limitations of VAP surveillance have led to the development and implementation by the NHSN of a new surveillance paradigm for ventilated patients that aims to assess ventilator-associated events (VAEs) (15–17). This algorithm identifies ventilator-associated conditions (VACs) and infection-related ventilator-associated conditions (IVACs) as entities for public reporting, using objective case definitions based on ventilator settings, antimicrobial prescriptions, temperature, and leukocyte counts that are amenable to automated implementation. The algorithm further defines possible and probable VAP for within-hospital quality monitoring (Table 1).
| Entity | Abbreviation | Brief Definition |
|---|---|---|
| Ventilator-associated event | VAE | Includes VAC, IVAC, possible VAP, and probable VAP |
| Ventilator-associated condition | VAC | New, sustained respiratory deterioration after a 2-d baseline period of stability or improvement on mechanical ventilation |
| Infection-related ventilator-associated condition | IVAC | VAC with evidence of infection (new antibiotics and inflammatory signs) |
| Ventilator-associated pneumonia (VAE) | VAE-VAP | IVAC with microbiological evidence of pneumonia (classified as possible or probable) |
| Ventilator-associated pneumonia (prospective) | PROSP-VAP | Evidence of VAP by prospective surveillance definition (classified as definite, probable, or possible VAP). Requires combination of clinical signs, radiographic and microbiological evidence |
This study aimed to assess the feasibility and validity of surveillance based on electronic implementation of the newly introduced VAE algorithm in a multicenter setting in the Netherlands. For this purpose, causes of VAC and IVAC signals were assessed to evaluate the face validity of VAE, VAE results were compared with ongoing prospective VAP registration, and mortality estimates were calculated. Sensitivity analyses were performed to evaluate several adaptions of the VAE algorithm and to assess the robustness of several approaches to electronic implementation.
Some results of this study have been previously reported in the form of abstracts (18, 19).
This cohort study was incorporated in the ongoing MARS (Molecular Diagnosis and Risk Stratification of Sepsis) project in the mixed intensive care units (ICUs) of two tertiary referral centers in the Netherlands (20). The institutional review board approved an opt-out consent method (IRB no. 10-056C). For the current study, we analyzed all adult patients who had received two or more consecutive days of mechanical ventilation (MV) between January 1, 2011 and July 1, 2012. Patients with do-not-resuscitate orders and patients ventilated in the prone position were included in the analysis. Patients on rescue MV (high-frequency ventilation, extracorporeal membrane oxygenation) were included, but the days on rescue ventilation were excluded from the analysis.
The main analysis was performed with the VAE algorithm implemented as specified by the current NHSN protocol, using electronically extracted data (minute-to-minute ventilator settings, antibiotic use, microbiology data, and clinical characteristics) (16). Ventilator-associated conditions are defined by either a greater than 3–cm H2O increase in daily minimum positive end-expiratory pressure (PEEP) or a greater than 20% increase in the fraction of inspired oxygen (FiO2). See the online supplement for details.
In sensitivity analyses, we assessed two sets of alternative implementations of the original algorithm. The first implementation aimed to identify more representative episodes of respiratory deterioration by varying the required increases in levels of PEEP and FiO2 to more than 5 cm H2O and more than 10%, respectively. In addition, the original algorithm does not incorporate the time patients are in spontaneous breathing trials when being weaned off the ventilator or when they are on nighttime ventilation. As this ignores the best condition of the patient for that day, a sensitivity analysis was performed that classifies respiratory deterioration leading to discontinuation of weaning trials as a VAC. This concept was implemented by setting the PEEP to room air conditions (PEEP = 0) if the patient was in a spontaneous weaning trial for more than 2.5 hours (10%) of the day (intermittent MV rule; Figure 1).
Figure 1. Left: Hypothetical example of a patient on intermittent ventilation under the original algorithm and the intermittent ventilation sensitivity analysis. If mechanical ventilation was interrupted for more than 2.5 hours (i.e., 10% of the day) then daily minimum values were set to room air conditions (PEEP = 0 and FiO2 = 21%). Right: Example of artifact filtering by both the 10th percentile (lowest 10% of daily measurements excluded) and the sustained settings rule (settings must be maintained for at least 60 consecutive minutes to qualify). The duration of the dip in PEEP on Day 4 was 80 minutes; the two dips on Day 5 were both 30 minutes. 10th perc = 10th percentile; PEEP = positive end-expiratory pressure; VAC = ventilator-associated condition.
[More] [Minimize]The second analysis aimed to evaluate the robustness of electronic implementation of the VAE algorithm, as the current guidelines do not precisely specify the requirements for electronic data capture. As opposed to the original implementation, which involves the use of minute-to-minute measurements, we evaluated the effect of sampling frequency by using hourly, manually validated measurements only, which may better reflect common availability of data in electronic form. Second, to assess the effect of possible artifacts in the data, we excluded outliers from the minute-to-minute measurements by selecting the minimum daily ventilator settings either after application of a 10th percentile cutoff (thus excluding the lowest 10% of measurements; Figure 1)—the 10th percentile rule—or by selecting the lowest setting maintained for at least one consecutive hour (sustained settings rule).
Patients were assessed daily for the development of VAP by dedicated well-trained observers with ongoing evaluation of interrater reliability (20); this was considered the reference standard (Table E1 in the online supplement). VAP was classified as definite, probable, or possible, and assessment was fully independent of the VAE algorithm.
To assess what conditions may lead to a VAC signal, all patients flagged by the VAC algorithm (10th percentile rule) from one hospital were manually reviewed by two independent physicians (P.M.C.K.K., M.S.M.v.M.), with consensus through discussion with a third reviewer if necessary. The diagnoses that we explicitly considered are defined in Table 2. Reviewers were unaware of patient IVAC and VAP status.
| Diagnosis | Identified by: | Frequency Identified | |
|---|---|---|---|
| VAC (n = 81) | IVAC (n = 31) | ||
| Pulmonary condition | |||
| Respiratory tract infection* | Imaging, cultures, initiation of antibiotics, clinical documentation | 23 (28) | 14 (45) |
| Atelectasis/sputum plug | Bronchoscopy, imaging, explicit clinical documentation | 12 (15) | 5 (19) |
| Pneumothorax | Chest tube placement | 2 (2) | 0 (0) |
| Pulmonary embolus | Therapeutic anticoagulants | 0 (0) | 0 (0) |
| Pleural effusion | Pleural drainage | 10 (12) | 3 (10) |
| Aspiration | Explicit clinical documentation | 5 (6) | 1 (3) |
| Extrapulmonary infection | |||
| New onset of SIRS/sepsis | Clinical documentation, microbiology, initiation of antimicrobials | 9 (11) | 5 (16) |
| Cardiac/circulatory | |||
| Volume overload | Initiation of diuretics | 23 (28) | 12 (39) |
| Heart failure | Inotropy or afterload reduction | 6 (7) | 4 (13) |
| Other | |||
| Abdominal distension | Laparotomy, ascites drainage | 9 (11) | 4 (13) |
| Acute neurological event | Imaging and clinical findings | 10 (12) | 3 (10) |
| No reason for VAC identified | 10 (12) | 2 (6) | |
All ICU admissions were included to assess population characteristics and concordance between surveillance methods. Concordance between prospective VAP surveillance, stratified by VAP likelihood, and the VAE algorithm was assessed, both at the ICU admission level and by using a window of ±2 days surrounding VAC, IVAC, or VAE-VAP. This is analogous to the original algorithm that defines IVAC and VAE-VAP on the basis of a 5-day window around VAC.
We assessed the effects of VAEs and VAP on ICU mortality, using competing-risk analyses. The direct effects of VAEs and VAP on outcome were estimated by the cause-specific hazard ratios (CSHRs) for each event (ICU discharge or ICU death). To evaluate the direct effect of the events on death, taking the competing event (discharge) into account, we calculated the subdistribution hazard ratio (SHR) (21). All analyses were adjusted for age, sex, Acute Physiology and Chronic Health Evaluation (APACHE) IV score, admission type (surgical vs. medical), and hospital. VAC, IVAC, VAE-VAP, and prospective VAP were included as time-dependent variables (22, 23).
All analyses were performed with SAS 9.2 (Cary, NC), R version 2.14 (www.r-project.org), and SPSS 20 (IBM Software, Armonk, NY).
During the study period, 3,473 patients were admitted to the ICU, of whom 2,080 patients (2,296 admissions) were ventilated for at least two consecutive calendar days. The median age was 62 years and 44% were surgical patients. Overall ICU mortality was 21% (Table 3).
| All Patients (n = 2,296) | VAC Positive (n = 152) | IVAC Positive (n = 65) | VAE-VAP Positive (n = 50) | PROSP-VAP Positive (n = 115) | |
|---|---|---|---|---|---|
| Number of events | — | 158 | 66 | 51 | 127 |
| Incidence (per 1,000 MV) | — | 10.0 | 4.2 | 3.2 | 8.0 |
| Age | 62 (50–72) | 60 (52–69) | 57 (49–66) | 57 (48–62) | 62 (49–71) |
| Male | 1,406 (61) | 102 (67) | 41 (63) | 32 (64) | 76 (66) |
| Comorbidities | |||||
| Cerebrovascular disease | 164 (7) | 9 (6) | 1 (2) | 0 (0) | 9 (8) |
| Congestive heart failure | 171 (7) | 14 (9) | 4 (6) | 3 (6) | 6 (5) |
| COPD | 247 (11) | 18 (12) | 8 (12) | 5 (10) | 16 (14) |
| Diabetes | 372 (16) | 26 (17) | 9 (14) | 8 (16) | 8 (7) |
| Malignancy | 334 (15) | 31 (20) | 12 (18) | 9 (18) | 16 (14) |
| Admission type | |||||
| Medical | 1,303 (57) | 91 (60) | 45 (69) | 37 (74) | 61 (53) |
| Surgical emergency | 567 (25) | 36 (24) | 8 (12) | 5 (10) | 37 (32) |
| Surgical elective | 426 (19) | 25 (16) | 12 (18) | 8 (16) | 17 (15) |
| Admission source | |||||
| Operating theater | 640 (27) | 37 (24) | 14 (21) | 9 (18) | 31 (27) |
| Emergency department | 810 (35) | 51 (34) | 18 (28) | 13 (26) | 48 (42) |
| Other ward | 647 (28) | 47 (31) | 25 (38) | 22 (44) | 23 (20) |
| Other ICU | 199 (9) | 17 (11) | 8 (12) | 6 (12) | 13 (11) |
| Readmission | 324 (14) | 22 (14) | 15 (23) | 14 (28) | 12 (10) |
| Primary specialty | |||||
| Cardiothoracic surgery | 347 (15) | 24 (16) | 10 (15) | 6 (12) | 25 (22) |
| Neurology or neurosurgery | 540 (24) | 18 (12) | 8 (12) | 8 (16) | 33 (29) |
| Other surgery | 648 (28) | 43 (28) | 17 (26) | 13 (26) | 20 (17) |
| Internal medicine | 707 (31) | 64 (42) | 29 (45) | 22 (44) | 36 (31) |
| Other, unknown | 54 (2) | 3 (2) | 1 (2) | 1 (2) | 1 (1) |
| APACHE IV | 75 (57–97) | 82 (65–105) | 79 (62–102) | 84 (56–102) | 77 (56–100) |
| Duration of MV, d | 4 (2–8) | 13 (8.5–27) | 15 (12–19) | 15 (10–29) | 14 (10–28) |
| Length of ICU stay, d | 5 (2–9) | 14 (10–29) | 19 (13–33) | 18 (12–33) | 15 (12–34) |
| Deceased in ICU | 476 (21) | 56 (37) | 18 (28) | 12 (24) | 39 (34) |
Using the original algorithm, 158 VACs were detected in 152 patients (10.0/1,000 d of MV). Most events (n = 149) were triggered by increasing PEEP settings, five by increased FiO2 levels, and four by both. There were 66 IVACs in 65 patients (4.2/1,000 MV days). All IVAC episodes that fulfilled the antibiotic exposure criteria also met the temperature and/or white blood cell count definition. There were 51 episodes of possible or probable VAP according to the VAE algorithm (3.2/1,000 MV days) in 50 patients and 127 episodes of possible, probable, or definite VAP were identified by prospective surveillance in 115 patients (8.0/1,000 MV days) (Table 3). A large fraction of VAC (and IVAC) events occurred on the third or fourth day of MV (46%). Of the 2,296 admissions, 108 did not achieve a baseline period of stability; of these, 60 died within 4 days of onset of ventilation. The incidence of VAE was higher in patients receiving seven or more days of MV compared with patients with less than 7 days of MV (VAC, 3.6 vs. 12.9 per 1,000 d of MV; IVAC, 0.6 vs. 5.8 per 1,000 d of MV). Rates of VAC, IVAC, VAE-VAP, and prospectively monitored VAP were comparable between both ICUs (Table E2).
The sensitivity of VAC for detection of (possible, probable, or definite) VAP was 33% (95% confidence interval [CI], 25–42%), with a positive predictive value of 25% (95% CI, 18–33%) (Table 4). For IVAC and VAE-VAP the sensitivities were 18% (95% CI, 12–27%) and 17% (95% CI, 10–25%), respectively, with positive predictive values of 32% (95% CI, 21–45%) and 38% (95% CI, 25–53%). When restricting concordance to the VAE window (VAP by prospective surveillance must have occurred within the 5-d window surrounding VAC), sensitivities for detecting VAP were 13, 9, and 6% for VAC, IVAC, and VAE-VAP, respectively, with positive predictive values of 10, 18, and 16%. Thus most episodes of VAC and IVAC were not temporally associated with VAP.
| Prospective VAP | ||||||
|---|---|---|---|---|---|---|
| Definite | Probable | Possible | Any | Absent | Total | |
| VAC | ||||||
| Present | 1 | 13 | 24 | 38 | 114 | 152 |
| Absent | 2 | 16 | 59 | 77 | 2,067 | 2,144 |
| IVAC | ||||||
| Present | 0 | 8 | 13 | 21 | 44 | 65 |
| Absent | 3 | 21 | 70 | 94 | 2,137 | 2,231 |
| VAE-VAP | ||||||
| Present | 0 | 7 | 12 | 19 | 31 | 50 |
| Absent | 3 | 22 | 71 | 96 | 2,150 | 2,246 |
| Total | 3 | 29 | 83 | 115 | 2,181 | 2,296 |
When restricting the reference standard to probable and definite VAP, events of VAC, IVAC, and VAE-VAP detected 44, 25, and 22% cases of VAP, respectively. Positive predictive values were 9, 12, and 11% (Table 4). Among the 35 episodes of probable or definite prospective VAP (32 patients) there were 25 episodes that did not fulfill the criteria for VAC, either because there was no baseline period of stability (n = 6) or no (sufficient) increase in ventilator settings (n = 19).
In a retrospective analysis of underlying clinical conditions, pneumonia, either VAP or preexisting pneumonia, appeared to be the most often observed cause of VAC (Table 2). The interrater agreement was moderate (kappa = 0.51). The positive predictive value of IVAC for all respiratory tract infections combined (VAP, hospital-acquired pneumonia, community-acquired pneumonia, and other lower respiratory tract infection) increased to 66% at the patient level and 36% when restricting concordance to the 5-day window (data not shown). The sensitivity of IVAC for detection of these combined conditions, however, was 7 and 3%, respectively.
As the majority of VACs was identified by increases in PEEP settings, the effect of setting a higher trigger (>5– vs. >3–cm H2O PEEP increase) was evaluated. This resulted in 51 VACs with a sensitivity of 7% and positive predictive value of 22% for the detection of VAP. Conversely, lowering the required increase in FiO2 (>10 vs. >20%) resulted in 224 episodes of VAC in 213 patients, with a sensitivity of 43% and positive predictive value of 23% for the detection of VAP.
Using the algorithm with handling of intermittent weaning, 241 patients with 261 VACs were identified, with only 101 episodes concordant with the original algorithm. This adapted algorithm had higher sensitivity (50%) and similar positive predictive value (24%) with respect to concordance with prospective surveillance.
Applying the 10th percentile rule in the original algorithm did not change the overall incidence of VAC (158 events in 152 patients), but only 117 of the detected VAC episodes were identical. The 10th percentile modification yielded similar results as the original VAC algorithm with respect to concordance with VAP surveillance (data not shown). Applying the sustained settings rule resulted in 157 patients with VAC with a sensitivity of 34% and a positive predictive value of 25%.
Using hourly (validated) measurements resulted in 152 episodes of VAC in 149 patients with a sensitivity of 30% and a positive predictive value of 23% for the detection of VAP at the patient level (13% sensitivity and 11% positive predictive value when examining episode-level concordance). Of the 152 episodes of VAC detected when using hourly (validated) measurements, 113 identical episodes were detected by the 10th percentile and the sustained settings rule.
Of the 158 episodes identified by the original algorithm, 104 (65%) were also identified by all other sensitivity analyses. Thus, although the total number of episodes identified was similar for all electronic implementations, there were differences between the algorithms in the types of episodes that were identified.
All types of VAE and VAP were significantly associated with an increased hazard of ICU death when taking into accounting the competing events process (Table 5). The association was strongest for VAC (time-averaged SHR, 3.9; 95% CI, 2.9–5.3), and lower for IVAC (SHR, 2.5; 95% CI, 1.5–4.1) and VAE-VAP (SHR, 2.0; 95% CI, 1.1–3.6). VAP according to prospective surveillance had the highest subdistribution hazard ratio for death (7.2; 95% CI, 5.1–10.3). Analysis of CSHRs revealed that VAC and VAP identified by prospective surveillance, but not the other VAEs, had a significant direct effect on the hazard of dying (Tables 5 and E3). In addition, all types of VAE resulted in a lower daily probability of being discharged from the ICU after the onset of VAE, exposing patients longer to a daily risk of dying in the ICU—thus the increased risk of dying in the ICU after VAE is mainly the result of prolonged stay in the ICU rather than the direct effect of VAE on mortality. There was no significant interaction between the type of ICU admission (medical or surgical) and the different conditions (VAC, IVAC, VAE-VAP, prospective VAP) regarding their effect on estimated associated mortality, although for VAE-VAP there was a trend for slightly higher associated mortality in surgical patients whereas the opposite was observed for VAP identified by prospective surveillance.
| VAC* | IVAC* | VAE-VAP* | PROSP-VAP* | |
|---|---|---|---|---|
| Crude mortality | 51/134 (38.1) | 17/56 (30.4) | 12/43 (27.9) | 39/104 (37.5) |
| CSHR death (95% CI) | 3.96 (2.43–6.45) | 0.98 (0.57–1.70) | 1.11 (0.60–2.05) | 2.00 (1.34–3.00) |
| CSHR discharge (95% CI) | 0.38 (0.26–0.56) | 0.47 (0.33–0.66) | 0.56 (0.30–1.05) | 0.45 (0.34–0.58) |
| SHR† (95% CI) | 3.92 (2.88–5.34) | 2.51 (1.52–4.12) | 1.99 (1.11–3.58) | 7.24 (5.09–10.3)‡ |
| Covariate (SHR) | ||||
| Age | 1.00 (0.99–1.00) | 1.00 (0.99–1.00) | 1.00 (0.99–1.00) | 1.00 (0.99–1.00) |
| Female (vs. male) | 1.01 (0.84–1.23) | 1.02 (0.86–1.26) | 1.04 (0.86–1.26) | 1.04 (0.86–1.24) |
| APACHE IV | 1.03 (1.02–1.03) | 1.03 (1.02–1.03) | 1.03 (1.03–1.03) | 1.03 (1.02–1.03) |
| Hospital | 1.32 (1.10–1.59) | 1.34 (1.11–1.61) | 1.34 (1.11–1.61) | 1.34 (1.11–1.61) |
| Medical admission (vs. surgical) | 1.06 (0.87–1.31) | 1.09 (0.86–1.30) | 1.06 (0.86–1.13) | 1.10 (0.90–1.35) |
Estimates of associated mortality for VAC identified using the various electronic implementations were SHR, 6.3 (95% CI, 4.8–8.4) for the 10th percentile rule; SHR, 5.2 (95% CI, 3.9–6.9) for the sustained settings rule; and SHR, 6.3 (95% CI, 4.7–8.5) for the hourly sampling scheme.
The development and implementation of a novel surveillance paradigm for ventilator-associated events exemplify the ongoing efforts toward reliable surveillance of health care–associated infections, and in particular of ventilator-associated complications and pneumonia. In this study the VAC, IVAC and VAP events had poor concordance with prospective VAP surveillance, especially when restricting the analysis to the 5-day window surrounding VAC events. Importantly, however, the VAE algorithm aims at identifying a broader range of ventilator-associated complications, and this is confirmed by findings from our retrospective review. Although a significant fraction of cases of VAC and IVAC could be attributed to pulmonary infections, albeit not limited to VAP, conditions such as volume overload, nonpulmonary infections, and a variety of other causes were also commonly implicated. Occurrence of a VAE was associated with an increased risk of death in ICU, however, not as strongly as the occurrence of VAP identified by prospective surveillance. Interestingly, both IVAC and VAP defined by the VAE algorithm were associated with lower likelihoods of ICU mortality than VAC, possibly indicating that other conditions than (pulmonary) infection were responsible for at least part of the associated mortality of VAC. Furthermore, although electronic implementation of the algorithm is feasible in two academic centers, subtle differences in the method of electronic implementation affect the events identified by the algorithm and their associated mortality. In the absence of detailed specifications, the algorithm was implemented using both detailed minute-to-minute data collection and a more practical hourly sampling scheme; interestingly, both data sources identified different episodes of VAE in different patients and had similarly poor concordance with prospective VAP surveillance.
Several other studies have also found moderate concordance of VAC with VAP occurrence at the patient level and some association of VAC with ICU mortality and length of stay (15, 24–26). These studies have also shown that VAC reflects a broader scope of clinical conditions than VAP alone (24, 27). The higher incidence of VAC in these studies may have resulted from differences in implementation of the algorithm or differences between study populations. The present study is the first to assess the concordance of IVAC and VAP identified by the VAE algorithm, comparing it with a preexisting prospective VAP surveillance and evaluating the reliability of electronic implementation.
The VAE algorithm has been implemented in the United States as a novel tool for surveillance and benchmarking of ventilator-associated complications, not limited to pneumonia. However, interpretation of VAE rates remains difficult and their usefulness for quality improvement has not yet been established. Several criteria could be helpful in evaluating the validity of this novel surveillance entity. Above all, the outcome of interest should be clinically relevant and measure all aspects of what it is intended to assess. Although this cannot be formally tested, the retrospective analysis of alternative diagnoses in the present study shows that VAC and IVAC measure a diversity of conditions, some of which may not be associated with (quality of) ventilation practices. Moreover, the detection of VAP—one of the major target conditions for the VAE algorithm—was poor. The large number of VACs occurring on the third or fourth day of ventilation could be representative of ongoing clinical deterioration as opposed to insufficient quality of care. Furthermore, IVACs appeared to detect respiratory infections not related to MV. Ideally, a surveillance method should also identify differences between groups of patients that differ in their underlying risk of developing the event of interest. Because this study was not aimed at detecting differences between hospitals and no interventions for VAP were implemented during the period of study, this could not be evaluated. In addition, a key aspect that remains to be assessed is the preventability of conditions identified by the VAE algorithm and their effect on patient-centered outcomes. Intervention trials evaluating quality improvement programs targeted at VAC and IVAC are needed to answer this question and results from the present study may help to improve the design of such studies. In a recently published post hoc analysis, VAP clinical practice guideline implementation—with increasing overall guideline compliance—modestly decreased VAC rates but not IVAC. No specific preventive measure alone was associated with lower VAC rates, and interventions targeted at VAE specifically were not evaluated (25).
In the present study, the great majority of VACs were identified by increases in PEEP as opposed to FiO2 increases. In our ICUs we have implemented the higher PEEP/lower FiO2 protocol from the ARDSNet guidelines (28). Using a decreased FiO2 cutoff improved sensitivity for the detection of VAP with a similar positive predictive value, which, in our setting, may therefore be a preferable alternative to the original algorithm. Importantly, the VAE algorithm was largely unknown during the study period and has not been adopted in the Netherlands, and therefore clinician decisions to change ventilator settings are expected to be fully independent of the new algorithm and in compliance with local protocols.
Introduction of the VAE algorithm was driven by a desire for more objective, efficient, and reliable measures of complication of MV (17) than the current manual assessment of VAP occurrence (11). Although the VAE algorithm uses objective criteria and is amenable to automated implementation, our sensitivity analyses demonstrate that small modifications in electronic implementation lead to important differences in events detected and estimates of associated mortality. In addition, previous studies have shown that manually collected variables are often different from those collected electronically (29), and thus care must be taken when comparing rates collected through manual surveillance with electronic surveillance. Because standardized electronic surveillance is not yet universally implemented, comparability across institutions using different electronic implementations or manual systems remains questionable and these concerns will need to be addressed in the future. Finally, from the perspective of benchmarking, additional developments regarding case mix correction are necessary before valid interhospital comparisons can be made.
This study has several limitations. The reference standard used in this study, prospective VAP surveillance, is inherently vulnerable to the disadvantages of VAP surveillance described previously. However, the assessment was done prospectively by multiple well-trained assessors and completely independent of the VAE algorithm. A prior study found that the agreement between raters was high overall (89%), but lower for VAP (35%) (20). However, as opposed to the retrospective study setting, the current process of prospective surveillance involves discussions among observers, discussions with (senior) clinicians in multidisciplinary meetings attended by critical care physicians and infection specialists, and continuous checks of data integrity. All prospective diagnoses were therefore made after consensus. In addition, the reliability found in the retrospective study was similar to previously reported studies (11). Furthermore, although the participating centers did not routinely perform bronchoalveolar lavage in patients with a clinical suspicion of VAP, this is representative of the diagnostic practices in most ICUs worldwide and thus adds to the generalizability of our findings. In our setting of selective decontamination of the digestive tract (SDD) all patients are screened for respiratory pathogens by collecting endotracheal specimens according to a standardized protocol on admission and at least twice weekly. Respiratory specimens were obtained from all patients suspected of VAP before the introduction of antimicrobial therapy and microbiological confirmation was present in all definite or probable cases and in two-thirds of possible cases. Second, the SDD regimen potentially lowers the risk of VAP (30, 31). However, we would not expect much better concordance and reliability in settings with higher VAP rates. Finally, the competing events analysis adjusted for age, sex, hospital, and baseline APACHE as confounders, but did not include time-varying confounders and thus some residual confounding may remain. However, the adjustment methods used were identical for all the entities compared.
This study shows (1) that detection of VAP by the novel surveillance paradigm for ventilator-associated events is poor, (2) that events detected as VAE represent a broad range of clinical conditions that may not be liable to preventive measures and thus do not necessarily represent quality of care, and (3) that small differences in electronic implementation can considerably affect incidence rates and associated mortality of events detected. More studies are needed to establish the clinical entities underlying VAEs, develop methods for case mix adjustment, and ensure that rates obtained from different institutions are comparable before considering these events as an established quality metric. Finally, given these important concerns, the VAE paradigm should not be used as the sole method of surveillance for VAP.
The authors acknowledge Cristian Spitoni for statistical advice and Mark de Jong for retrieving mechanical ventilator data.
Members of the MARS Consortium: Friso M. de Beer, M.D., and Lieuwe D. J. Bos, M.D. (Department of Intensive Care Medicine, Academic Medical Center, University of Amsterdam, Amsterdam, the Netherlands); Jos F. Frencken, M.D. (Department of Intensive Care Medicine and Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands); and Roosmarijn T. M. van Hooijdonk, M.D., Mischa A. Huson, M.D., Laura R. A. Schouten, M.D., Marleen Straat, M.D., Lonneke A. van Vught, M.D., Maryse A. Wiewel, M.D., Esther Witteveen, M.D., Gerie J. Glas, M.D., and Luuk Wieske, M.D. (Department of Intensive Care Medicine, Academic Medical Center, University of Amsterdam).
| 1. | Association for Professionals in Infection Control and Epidemiology (APIC). HAI reporting laws and regulations. 2011 [accessed 2014 Mar]. Available from: http://www.apic.org/Resource_/TinyMceFileManager/Advocacy-PDFs/HAI_map.gif |
| 2. | Klevens RM, Edwards JR, Richards CL Jr, Horan TC, Gaynes RP, Pollock DA, Cardo DM. Estimating health care–associated infections and deaths in U.S. hospitals, 2002. Public Health Rep 2007;122:160–166. |
| 3. | Panzer RJ, Gitomer RS, Greene WH, Webster PR, Landry KR, Riccobono CA. Increasing demands for quality measurement. JAMA 2013;310:1971–1980. |
| 4. | Edwards JR, Peterson KD, Mu Y, Banerjee S, Allen-Bridson K, Morrell G, Dudeck MA, Pollock DA, Horan TC. National Healthcare Safety Network (NHSN) report: data summary for 2006 through 2008, issued December 2009. Am J Infect Control 2009;37:783–805. |
| 5. | van der Kooi TI, Manniën J, Wille JC, van Benthem BH. Prevalence of nosocomial infections in The Netherlands, 2007–2008: results of the first four national studies. J Hosp Infect 2010;75:168–172. |
| 6. | Melsen WG, Rovers MM, Groenwold RH, Bergmans DC, Camus C, Bauer TT, Hanisch EW, Klarin B, Koeman M, Krueger WA, et al. Attributable mortality of ventilator-associated pneumonia: a meta-analysis of individual patient data from randomised prevention studies. Lancet Infect Dis 2013;13:665–671. |
| 7. | Skrupky LP, McConnell K, Dallas J, Kollef MH. A comparison of ventilator-associated pneumonia rates as identified according to the National Healthcare Safety Network and American College of Chest Physicians criteria. Crit Care Med 2012;40:281–284. |
| 8. | Uçkay I, Ahmed QA, Sax H, Pittet D. Ventilator-associated pneumonia as a quality indicator for patient safety? Clin Infect Dis 2008;46:557–563. |
| 9. | Michetti CP, Fakhry SM, Ferguson PL, Cook A, Moore FO, Gross R; AAST Ventilator-Associated Pneumonia Investigators. Ventilator-associated pneumonia rates at major trauma centers compared with a national benchmark: a multi-institutional study of the AAST. J Trauma Acute Care Surg 2012;72:1165–1173. |
| 10. | Horan TC, Andrus M, Dudeck MA. CDC/NHSN surveillance definition of health care–associated infection and criteria for specific types of infections in the acute care setting. Am J Infect Control 2008;36:309–332. |
| 11. | Klompas M. Interobserver variability in ventilator-associated pneumonia surveillance. Am J Infect Control 2010;38:237–239. |
| 12. | Fàbregas N, Ewig S, Torres A, El-Ebiary M, Ramirez J, de La Bellacasa JP, Bauer T, Cabello H. Clinical diagnosis of ventilator associated pneumonia revisited: comparative validation using immediate post-mortem lung biopsies. Thorax 1999;54:867–873. |
| 13. | Klompas M. Eight initiatives that misleadingly lower ventilator-associated pneumonia rates. Am J Infect Control 2012;40:408–410. |
| 14. | Lin MY, Bonten MJ. The dilemma of assessment bias in infection control research. Clin Infect Dis 2012;54:1342–1347. |
| 15. | Klompas M, Khan Y, Kleinman K, Evans RS, Lloyd JF, Stevenson K, Samore M, Platt R; CDC Prevention Epicenters Program. Multicenter evaluation of a novel surveillance paradigm for complications of mechanical ventilation. PLoS One 2011;6:e18062. |
| 16. | National Healthcare Safety Network (NHSN). The National Healthcare Safety Network device-associated module: ventilator-associated event protocol. Atlanta, GA: Centers for Disease Control and Prevention; July 2013. |
| 17. | Magill SS, Klompas M, Balk R, Burns SM, Deutschman CS, Diekema D, Fridkin S, Greene L, Guh A, Gutterman D, et al. Developing a new, national approach to surveillance for ventilator-associated events. Crit Care Med 2013;41:2467–2475. |
| 18. | Klein Klouwenberg PM, Van Mourik MS, Ong DS, Cremer OL, Bonten MJ. Validation of a new surveillance algorithm for ventilator associated pneumonia [abstract]. European Society of Clinical Microbiology and Infectious Diseases (ECCMID): Berlin; 2013. |
| 19. | Van Mourik MS, Klein Klouwenberg PM, Ong DS, Schultz MJ, Horn J, Cremer OL, Bonten MJ. Validation and assessment of the new surveillance paradigm for ventilator-associated events [abstract]. Antimicrob Resist Infect Control 2013;2:O64. |
| 20. | Klein Klouwenberg PM, Ong DS, Bos LD, de Beer FM, van Hooijdonk RT, Huson MA, Straat M, van Vught LA, Wieske L, Horn J, et al. Interobserver agreement of Centers for Disease Control and Prevention criteria for classifying infections in critically ill patients. Crit Care Med 2013;41:2373–2378. |
| 21. | Fine JP, Gray RJ. A proportional hazards model for the subdistribution of a competing risk. J Am Stat Assoc 1999;94:496–509. |
| 22. | Beyersmann J, Gastmeier P, Wolkewitz M, Schumacher M. An easy mathematical proof showed that time-dependent bias inevitably leads to biased effect estimation. J Clin Epidemiol 2008;61:1216–1221. |
| 23. | Beyersmann J, Allignol A, Schumacher M. Competing risks and multistate models with R. New York: Springer; 2012. |
| 24. | Hayashi Y, Morisawa K, Klompas M, Jones M, Bandeshe H, Boots R, Lipman J, Paterson DL. Toward improved surveillance: the impact of ventilator-associated complications on length of stay and antibiotic use in patients in intensive care units. Clin Infect Dis 2013;56:471–477. |
| 25. | Muscedere J, Sinuff T, Heyland DK, Dodek PM, Keenan SP, Wood G, Jiang X, Day AG, Laporta D, Klompas M. The clinical impact and preventability of ventilator-associated conditions in critically ill mechanically ventilated patients. Chest 2013;144:1453–1460. |
| 26. | Ding S, Kilickaya O, Senkal S, Gajic O, Hubmayr RD, Li G. Temporal trends of ventilator-associated pneumonia incidence and the effect of implementing health-care bundles in a suburban community. Chest 2013;144:1461–1468. |
| 27. | Klompas M, Magill S, Robicsek A, Strymish JM, Kleinman K, Evans RS, Lloyd JF, Khan Y, Yokoe DS, Stevenson K, et al.; CDC Prevention Epicenters Program. Objective surveillance definitions for ventilator-associated pneumonia. Crit Care Med 2012;40:3154–3161. |
| 28. | ARDSNet. Mechanical ventilation protocol summary of the NIH NHLBI ARDS Clinical Network [accessed March 2014]. Available from: http://www.ardsnet.org/system/files/Ventilator%20Protocol%20Card.pdf |
| 29. | Bosman RJ, Oudemane van Straaten HM, Zandstra DF. The use of intensive care information systems alters outcome prediction. Intensive Care Med 1998;24:953–958. |
| 30. | Liberati A, D’Amico R, Pifferi S, Torri V, Brazzi L, Parmelli E. Antibiotic prophylaxis to reduce respiratory tract infections and mortality in adults receiving intensive care. Cochrane Database Syst Rev [serial on the Internet]. 2009 [accessed March 2014]; 4:CD000022. Available from: http://summaries.cochrane.org/ |
| 31. | Bonten MJ. Healthcare epidemiology: ventilator-associated pneumonia: preventing the inevitable. Clin Infect Dis 2011;52:115–121. |
*P.M.C.K.K. and M.S.M.v.M. contributed equally to this work as co–first authors.
A complete list of members of the MARS Consortium may be found before the beginning of the References.
Supported by the Center for Translational Molecular Medicine (http://www.ctmm.nl), project MARS (grant 04I-201). M.J.M.B. has received research funding from the Netherlands Organization of Scientific Research (NWO Vici 918.76.611). The sponsors did not play a role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.
Author Contributions: Designed the study: P.M.C.K.K., M.S.M.v.M., O.L.C., and M.J.M.B. Performed the analyses: P.M.C.K.K. and M.S.M.v.M. Collected the data: P.M.C.K.K., M.S.M.v.M., D.S.Y.O., and members of the MARS Consortium. Drafted the manuscript: P.M.C.K.K. and M.S.M.v.M. Critically reviewed the manuscript: P.M.C.K.K., M.S.M.v.M., D.S.Y.O., J.H., M.J.S., O.L.C., M.J.M.B., and all members of the MARS Consortium. M.S.M.v.M. and P.M.C.K.K. had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis. All authors declare that no conflicts of interest exist.
This article has an online supplement, which is accessible from this issue’s table of contents online at www.atsjournals.org
Originally Published in Press as DOI: 10.1164/rccm.201307-1376OC on February 5, 2014
Author disclosures are available with the text of this article at www.atsjournals.org.
