The overall goal was to create a short but inclusive test that would measure essential and evidence-based knowledge necessary for a senior medical resident to provide effective care for a mechanically ventilated patient. Eight board-certified medical intensivists with extensive experience in practicing and teaching critical care medicine in an academic setting, as well as in publishing on mechanical ventilation and medical education topics, were involved in test writing. This group aimed to compile a case-based, multiple choice examination written in the style of the American Board of Internal Medicine certification test. The group intended the test to address five specific areas of knowledge relevant to mechanical ventilation: (1) cardiopulmonary physiology, (2) indications for mechanical ventilation, (3) diagnosis and management of mechanical ventilator complications, (4) use of different ventilator modes and settings, and (5) interpretation of data from mechanical ventilators. The test consisted of four case vignettes featuring patients with commonly encountered potential indications for mechanical ventilation such as chronic obstructive pulmonary disease, asthma, acute respiratory distress syndrome (ARDS), and pneumonia. Multiple choice questions with four single-response options followed each case presentation.

Twenty-five questions were initially generated and circulated by e-mail among the expert group members for comment. Four questions were dropped that related to issues not adequately supported by the medical literature, and two others were dropped because they were felt to be of lesser importance to a nonintensivist. All but five items remaining were changed on the basis of comments by group members, compiled by the primary author, and recirculated among group members three times before a pilot test consisting of 19 items was agreed on unanimously. A questionnaire was appended to the test to gather demographic information about the participants and to allow residents to report their perceptions about their training in mechanical ventilation as well as satisfaction with training.

We aimed to recruit about 30 to 40 residency programs for this study, seeking to include programs diverse in size, academic affiliation, and regional location. Our goal was to understand the outcomes of education in mechanical ventilation at a full range of programs.

Between December 2000 and January 2001, we sent e-mail messages to the 291 internal medicine training programs with functional e-mail addresses on the Accreditation Council of Graduate Medical Education's Web site (14). These messages described the study protocol and requested the participation of each program in the study. A total of 137 programs responded, of which 82 (28%) agreed to participate in the study. Two months later, we polled program directors again to assess expected compliance with the study protocol and feasibility for the program. At this point 12 failed to respond, 24 either could not guarantee full participation in the study protocol or expected a significant number of residents to be off site, and 8 declined to participate. We then chose a sample of 26 programs from the 36 remaining to assemble as diverse a group of programs as possible yet equally distributed in terms of geographic region, small versus large numbers of residents in the program, urban versus suburban or rural setting, and university versus community hospital affiliation.

Before testing a larger sample of residents, we sought to determine whether our questionnaire had evidence of validity (15), that is, if it truly measured mechanical ventilator knowledge of senior residents. The involvement of mechanical ventilation experts in the test development fulfilled assumptions for content validity, or the degree of appropriateness of test items included.

To evaluate more rigorous aspects of validity, the initial test was administered by residency or fellowship program directors at 5 university medical centers randomly selected from the 36 that agreed to participate. A total of 132 participants returned tests, including 103 internal medicine residents of all three levels of training, 19 pulmonary and critical care medicine fellows, and 10 attending physicians trained in pulmonary and critical care medicine. The anonymous tests took about 45 minutes to finish and were completed as a proctored group in most cases. The mean percentage of correct answers on the pilot test ranged from 67% (SD, 17%) for postgraduate first-year residents to 95% (SD, 6%) for attending physicians.

We evaluated the criterion validity of the questionnaire by testing for an empirical association between test scores and duration of training. Using one-way analysis of variance testing, we found that test scores increased significantly overall with year of training (p < 0.0001) from interns to senior residents. Last, we aimed to demonstrate evidence of construct validity, the theoretical assessment of validity based on the strength of relationships between test scores and other measured variables (15). We hypothesized that test scores would have high correlation with year of training (r > 0.6) but low correlations (r < 0.15) with testing site. Because our test was designed to assess knowledge needed by a senior resident to manage mechanically ventilated patients, we also hypothesized that test scores would increase significantly between each year of training, but not between fellows and attending physicians. These relationships behaved as expected, as we found Spearman correlation coefficients of 0.57 (p < 0.0001) and 0.03 (p = 0.49) between test scores and duration of training and test site, respectively. Using two-sample t tests, we also showed that there were significant differences in scores between each year of training (all p < 0.001), although there were no significant differences between the scores of fellows and attending physicians (p = 0.35). We made no further changes to the test (see resident test in online supplement).

A total of 26 residency programs with 347 eligible senior residents not involved in the preliminary test participated in the final testing phase between April and July 2001. We tested only senior residents as close to the end of their training as possible, because we wished to measure the success of training programs in providing adequate education in mechanical ventilation over the course of a resident's entire 3 years of training. Program directors or chief medical residents administered the anonymous questionnaires, with no time limit, to senior residents at a single group gathering whenever possible. Most tests were proctored by a program director or chief resident, who then returned them by mail to the main study site. Program directors completed questionnaires before the residents' anonymous answer sheets were collected.

The main outcome variable was percentage of correct answers on the mechanical ventilator test. Independent variables measured included characteristics of participants (satisfaction with training, perceived barriers to learning, and experiences in the ICU), residency programs (methods of resident evaluation, ICU organization, type of educational program in mechanical ventilation), and the perceptions of both residents and their program directors about the adequacy of trainee knowledge. Response choices for items addressing training perceptions included “strongly agree,” “agree,” “disagree,” and “strongly disagree,” although some included binary (“yes” and “no”) responses. ICU organization was characterized as “open” if patients were monitored primarily by resident teams with both ICU and ward duties and “closed” if teams provided care solely for ICU patients.

Univariate descriptive statistics were calculated for participants. We also used two-sample t tests or one-way analysis of variance tests to compare and report test scores by participant and program-level characteristics. Testing also was done using Wilcoxon rank sum and Kruskal–Wallis tests, although the results were similar to those obtained by parametric methods. Pearson χ² testing was used to compare dichotomous variables. Spearman correlations between variables found in bivariate analyses to have significant (p < 0.05) or borderline significant (p < 0.25) relationships with the main outcome variable, test score, were determined before modeling to examine for potential collinearity. Variables describing university versus community program setting and number of hospital beds were dropped because of strong correlations (r = −0.68 and 0.53, respectively) with number of residents, which itself was found to be most significantly related to test scores in earlier t tests (p = 0.003). Next, a multiple linear regression model was fit with the remaining important independent variables to determine which resident and program characteristics were associated with better test scores. Partial F tests were used sequentially to drop variables that did not contribute significantly to the final model. Correction for intraclass correlation related to potential residency program site effect was performed by variance estimate adjustment in the linear regression model (16). Data analysis was performed using Stata version 7.0 (Stata, College Station, TX). The Institutional Review Board of the University of North Carolina (Chapel Hill, NC) approved our research protocol.

Characteristics of the senior residents and their residency programs are shown in Table 1

The mean test score for senior residents on the mechanical ventilation test was 74% correct (SD, 14%; range, 37 to 100%). Ten percent of residents answered fewer than half of the questions correctly and more than one-third answered fewer than 70% correctly. Mean combined test scores by residency program ranged from 46 to 87% correct.

The performance of residents on specific test items is shown in Table 2

Associations between test scores and characteristics of both residents and their residency programs are shown in Table 3

Multiple linear regression modeling demonstrated better scores for residents in closed versus open ICUs (76 versus 71% correct, p = 0.033), for those with greater perceived knowledge level (79 versus 71% correct, p = 0.001), and for those who were U.S. versus international medical graduates (75 versus 69% correct, p = 0.001) after adjustment for program site effect and cluster sampling.

Using Spearman correlations, we found that resident satisfaction was strongly associated with awareness of learning objectives (r = 0.89, p < 0.0001), perceived adequacy of ventilator skills (r = 0.67, p < 0.0001), and belief that time for instruction was adequate (r = 0.57, p < 0.0001). Lack of learning objectives by programs had a significant negative correlation with resident contentment (r = −0.50, p < 0.0001). Satisfaction of program directors with ventilator training offered (r = 0.14, p = 0.03), number of attending physicians (r = 0.09, p = 0.16), and confidence of program directors in the ventilator skill of residents (r = 0.02, p = 0.98) were not related to resident satisfaction.

Questionnaire responses for both residents and program directors are summarized in Table 4

Our findings should be placed in the context of a growing burden of critical illness and a future deficit of specialists in critical care medicine. Considering the increased utilization of intensive care by society as a whole (8) and by older patients in particular (8, 12), a basic understanding of how to use the most common mode of organ support, mechanical ventilation, should be considered an essential component of an internist's knowledge. Although we do not advocate such a detailed understanding of ventilator issues as that required by subspecialists, there are several cornerstones of management that should be emphasized by residency programs to physicians in training so that the management of critically ill patients can be optimized. First, the tidal volume of a patient with ARDS should be targeted to 6 ml/kg predicted body weight rather than the more traditional 12 ml/kg (1). Using this lower tidal volume will save 1 life for every 10 patients treated in this manner. Second, patients who are stabilized or improving on the ventilator should be assessed every day for their ability to wake up and spontaneously breathe without ventilatory assistance (3, 17, 18). These approaches to liberation from the ventilator are now considered components of the accepted standard of care (5, 19). Also, using noninvasive positive pressure ventilation to manage selected patients with acute exacerbations of chronic obstructive pulmonary disease saves lives and reduces utilization of resources (7, 20, 21). Considering these examples from the more recent literature, the implications of our survey are that increasing the basic mechanical ventilator knowledge of the nearly 2,800 annual graduating internists (22) in the United States would enhance the likelihood of improving patient outcomes, realizing considerable cost savings, and saving lives.

We believe that the most valuable and efficient step in improving resident training in mechanical ventilation may be the use of specific evidence-based learning objectives. Emphasizing key objectives would not require increasing the quantity of ICU rotations, but would likely improve the quality of these rotations for this highly motivated group already burdened with numerous training requirements. Our finding that both test scores and satisfaction were related to awareness of learning objectives, but not number of months of ICU training, further supports this suggestion. Unfortunately, few programs we studied used learning objectives in their critical care instruction. However, programs have received little guidance in developing evidence-based mechanical ventilation or critical care learning objectives from national educational, internal medicine, or critical care organizations.

An essential aspect of improving knowledge of residents should include the close monitoring of educational outcomes with one or a combination of evaluation tools (23, 24). Our suggestions echo those of the Accreditation Council of Graduate Medical Education Outcome Project, a long-term initiative begun in 1999 to emphasize the use of learning objectives, to increase reliance on dependable methods of competency assessment, and to use outcome results to improve both resident performance and the quality of instruction (25). However, almost all the programs in our study relied only on the opinion of an attending physician for evaluation of resident skills, although faculty members often perform poorly in identifying residents with inadequate clinical competence (26). Half of the program directors in our study reported that the Outcome Project had not influenced their desire to change their educational programs in some way.

The role of the teaching physician is important to consider in improving education. Trainees highly valued the instruction of attending physicians and fellows. This finding supports the recommendation by the Residency Review Committee of the American Board of Internal Medicine that critical care specialists be available for resident supervision and training in the ICU. However, the poor performance of residents on items relating to more recent landmark clinical studies may indicate that attending physicians and fellows have not widely implemented these newer, important changes in their standard practice (27). Clear learning objectives for mechanical ventilation based on good evidence that are written and disseminated by professional peer organizations will guide not only the education of trainees, but will also inform instructors who are unaware of the more recent literature.

Program-level interventions that may improve knowledge could include a brief “hands on” course in mechanical ventilation management at the beginning of the academic year or a specific ICU rotation. This would provide a low-stress opportunity for trainees to become familiar with ventilators as well as dedicated time to review important learning objectives. Others have demonstrated that a similar 1-day annual course in ICU infection control practices for house staff was not only acceptable to residents and staff but improved patient outcomes and reduced costs due to adverse events (28). We recognize that the competing demands of training are significant, and that knowledge deficits persist in other areas of internal medicine training (29–31). Therefore, extra training in this one topic may not be feasible. This is another reason to emphasize clearly stated learning objectives during an ICU rotation. These objectives will improve the focus of instruction in the ICU and also guide residents in independent learning without adding additional burdens to the program, its instructors, and its trainees.

Finally, it may be helpful for residency programs to further consider the characteristics of residents associated with worse educational outcomes in our study, such as training in an ICU with a closed organization, international medical graduate (IMG) status, and resident-perceived inadequacy of knowledge. Residents who train in open ICUs may receive less focused critical care instruction and have less exposure to specialists because of divided patient care responsibilities between hospital wards and the ICU. A previous study indicated that residents who trained in a closed ICU setting felt more experienced and had a higher confidence level when managing ICU patients compared with residents who trained in open ICUs (32). The slightly worse performance of IMGs was surprising because of superior scores achieved by IMGs relative to U.S. graduates on the Internal Medicine In-Training Examination (33). It is possible that program-level factors for which we were unable to control in our analysis could explain this observation more completely. For example, IMGs may have had less exposure to critical care medicine before residency than U.S. graduates. Residents accurately predicted their own knowledge deficits, an observation that reproduces findings from a study of physician knowledge of pulmonary artery catheters (31). This may be reassuring, assuming that those with lower levels of perceived knowledge would consult others when contemplating difficult management decisions. It is important to note that the differences between these subgroups were typically the equivalent of one more incorrect test item. Therefore, we believe that these findings should not distract the reader from the main message of this study: residents from all subgroups fared poorly on evidence-based items that have immediate impact on patient mortality and costs.

Limitations of our study include a nonrandom sampling of residents, a factor that could have biased our results. However, we believe that test scores would likely favor higher values as a result, because residents with poor knowledge of mechanical ventilation would likely have refused participation. Similarly, training programs that lacked strong critical care instruction likely would have been less willing to participate. We recognize that although written tests such as ours are easily administered, they may overestimate knowledge level relative to clinical skills as assessed by objective structured clinical examinations or computer-controlled patient simulators (34). However, we believe that this observation simply highlights the concerning nature of our findings. Although we admit that an incorrect item response may not reflect inadequate clinical knowledge of the relevant concept, the incorrect responses of many residents were not just “close misses” on our test. Nearly all residents, for example, who incorrectly identified an appropriate tidal volume to give a patient with ARDS would have provided a tidal volume nearly double what is accepted as standard of care. Finally, even though we sampled 362 residents from a diverse group of 31 residency programs of varying size and academic affiliation found in 20 different states, we recognize that our findings may not be generalizable to all internal medicine residency programs and internal medicine residents in the United States.

In summary, we found that during their training many senior internal medicine residents are not gaining important evidence-based knowledge needed to provide effective care for patients who require mechanical ventilation. Instructors in critical care medicine should update their curriculum to reflect standard evidence-based practice. Residency programs should improve methods used to evaluate their educational effectiveness and develop specific mechanical ventilator learning objectives in concert with national graduate medical education, internal medicine, and critical care medicine organizations to achieve improved educational outcomes.