.png){kind=logo}
Abstract
Background: Entrustable professional activities (EPAs) define the essential tasks expected of subspecialists in unsupervised practice. Although EPAs have been piloted in some programs, their use for summative assessment of pediatric pulmonology fellows for graduation has not been studied.
Objective: To determine the minimum level of supervision that pediatric pulmonary program directors (PDs) require of their fellows for graduation and compare it with the minimum level of supervision they expect for a practicing subspecialist for the five pediatric pulmonology EPAs.
Methods: Using a modified Delphi approach, we developed supervision scales for the five pediatric pulmonology EPAs and conducted a national survey of pediatric pulmonary PDs in the United States through the Subspecialty Pediatric Investigators Network between April 2017 and August 2017.
Results: Forty-six pediatric pulmonary PDs completed the survey, representing a response rate of 85%. The majority did not require fellows to be trusted to practice without supervision for graduation for any of the five EPAs (level 5); the median minimum level of supervision they required was 4, equating to indirect supervision for complex cases. The minimum level for graduation, defined by consensus as the level of supervision for which no more than 20% of PDs would want the level to be lower to allow a fellow to graduate, was 3, which corresponded to requiring supervision for both simple and complex cases. There was a statistically significant difference between the minimum level of supervision deemed necessary by PDs for graduation and for practice as a subspecialist for each of the EPAs.
Conclusion: Most pediatric pulmonary PDs reported that they would graduate fellows who may still require indirect supervision for the five pediatric pulmonology EPAs. The findings suggest a need for stakeholders to reevaluate the structure and outcomes of training programs and ensure support for pediatric pulmonologists in their early practice period.
Entrustable professional activities (EPAs) have been developed to address the challenges in competency-based assessment in medical education (1–3). EPAs are measurable units of observable work, and “entrustment” requires that the competencies needed to complete that work be integrated for safe and effective practice without supervision (1, 2). Collectively, the EPAs define the tasks expected of pediatric subspecialists in unsupervised practice (3). Seven common EPAs and scales were developed, and validity evidence accrued for the pediatric subspecialties (4–6). In 2015, five EPAs were developed for pediatric pulmonology by a taskforce working in collaboration with the American Board of Pediatrics (ABP) (1, 4). Although EPAs are being piloted in a number of training programs for formative assessment and curriculum development (7–9), their use as summative assessments for advancement and graduation is limited.
Pediatric pulmonary program directors (PDs) are responsible for overseeing the clinical and scholarly training of fellows and play an important role in the decision to graduate their fellows. Although PDs receive assessments from clinical competency committees, they are responsible for providing a final evaluation, attesting that fellows are competent and can practice autonomously (10). However, their expectations for the performance of graduating fellows are unclear.
We developed supervision scales for the pediatric pulmonology EPAs and then conducted a national survey of pediatric pulmonary PDs in the United States to determine the minimum level of supervision that they require fellows to achieve for graduation and compared it with the minimum level they expect of practicing subspecialists to provide safe and effective patient care. We hypothesized that pediatric pulmonary PDs would not require fellows to be trusted to practice without supervision for graduation and that there would be a gap between the minimum level of supervision they deem necessary for graduation and the minimum level they deem necessary for practice. The study holds important implications for the goals and structure of training programs as well as the support of early practicing pediatric pulmonologists.
Using a modified Delphi approach, the steering committee of the Subspecialty Pediatrics Investigator Network (SPIN) (11), an educational research network, adapted the supervision scales to the pediatric pulmonology EPAs with input from members of the Pediatric Pulmonary Training Directors Association in 2017 (Table 1). The SPIN steering committee is composed of up to two representatives from each of the 14 pediatric subspecialties approved by the Accreditation Council for Graduate Medical Education (ACGME) and for which the ABP offers certification, as well as representatives from the Council of Pediatric Subspecialties, ABP, and Association of Pediatric Program Directors Fellowship Executive Committee and Longitudinal Educational Assessment Research Network. All members of SPIN and Pediatric Pulmonary Training Directors Association have graduate medical education experience.
| EPA | Level of Supervision Scale | ||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
| ||||||||||||||||||||||||||||||
We sent an anonymous survey (see Text E1 in the data supplement) through SPIN using LimeSurvey between April and August 2017. This survey was part of a larger study that included fellowship PDs from 14 ACGME-accredited fellowships with ABP certification. Support for content validity was obtained by reviewing the literature and using expert review by SPIN steering committee members. We pilot-tested the survey with three pediatric pulmonary faculty members who had graduate medical education experience but were not current PDs, revising it on the basis of their feedback. The study was reviewed by the University of Illinois at Chicago Institutional Review Board and was found to be exempt. We sent the original invitation and then at least four reminder emails to prospective participants via LimeSurvey. Nonresponders were then contacted personally via e-mail by the SPIN pulmonary representative three times and asked to complete the survey.
The title and functions of each of the five pediatric pulmonology EPAs were described in the survey (see Text E1 in the data supplement); questions for each included 1) “For this EPA, what do you believe is the MINIMUM level of supervision a fellow must achieve to successfully complete fellowship?” 2) “For this EPA, if a fellow did not achieve at least this minimal level of supervision, would you still allow him/her to graduate?” with choices of “yes” or “no,” and 3) “For this EPA, what is the LOWEST level in which you would consider that a practicing subspecialist (and not necessarily a trainee) should be able to perform most of the activities described above resulting in a safe and effective outcome?” To ensure a shared mental model, we explained the concept of entrustment in choosing the supervision level, specifically that the PD does not necessarily have to see a fellow perform the functions to trust them to do so. In addition, we collected information from PDs about the number of fellows in the program, duration of program directorship, understanding of EPAs, use of EPAs, and participation in the previous SPIN study (6).
Data were described as number and percentage for categorical survey items and as median and percentiles. After discussions with experts in both medical education and pediatric subspecialties, we set the minimum level of supervision required for graduation at the level that PDs required for graduation or one level below if the PDs reported in the second question that they would allow the fellow to graduate even if the fellow did not achieve the expected level. Similarly, by expert consensus and to align with previously published residency EPA performance data and standards (12), we set the standard for the minimum level for graduation and the minimum level for practice as the level of supervision at which no more than 20% of PDs would want the level to be lower for each EPA. We tested the differences in level of supervision between graduation and practice for each EPA using Wilcoxon signed-rank tests. We measured the correlation between key variables with graduation and entrustment levels using Spearman’s ρ. A P value of less than 0.05 was considered significant. Data analyses were conducted using R 3.2.2 (13).
Forty-six of 54 pediatric pulmonary PDs completed the survey, a response rate of 85%. The median duration of program directorship was 5 years. Of the PDs, 27 (59%) had zero to three fellows, 16 (35%) had four to six fellows, and three (6.5%) had more than six fellows. Of the PDs, nine (20%) and 21 (46%) participated in the previous SPIN study (6) and were using EPAs, respectively; 24 (52%) and 20 (44%) rated their understanding of EPAs as basic or in depth, respectively, whereas <3% were either unfamiliar or expert.
The minimum level of supervision that PDs required for graduation for each of the five pediatric pulmonology EPAs are shown in Table 2 (see Table E1 for unadjusted data). The majority of PDs did not require fellows to achieve a minimum supervision level of 5, or unsupervised practice, for graduation for any of the EPAs. Of the PDs who did not require unsupervised practice for graduation, most would allow fellows to graduate with indirect supervision for complex cases; however, for “communicate a new diagnosis,” 54% of them would allow fellows to graduate even if indirect supervision was required for both simple and complex cases. The median minimum level of supervision required by PDs for graduation was 4, requiring indirect supervision for complex cases for all five EPAs (Table 3). However, the minimum level for graduation was 3, indicating that indirect supervision was required for both simple and complex cases, for all EPAs.
| Level of Supervision | EPA | ||||
|---|---|---|---|---|---|
| Manage Acute Complex Respiratory Disease [n (%)] | Manage Chronic Respiratory Disease [n (%)] | Communicate a New Diagnosis [n (%)] | Manage Respiratory Equipment [n (%)] | Perform Procedures [n (%)] | |
| Graduation* | |||||
| 1 | 0 (0) | 0 (0) | 2 (4) | 0 (0) | 0 (0) |
| 2 | 2 (4) | 4 (9) | 4 (9) | 2 (4) | 5 (11) |
| 3 | 10 (22) | 9 (20) | 15 (33) | 12 (26) | 7 (15) |
| 4 | 29 (63) | 28 (61) | 18 (39) | 26 (57) | 25 (54) |
| 5 | 5 (11) | 5 (11) | 7 (15) | 6 (13) | 9 (20) |
| Practice | |||||
| 1 | 0 (0) | 0 (0) | 0 (0) | 0 (0) | 0 (0) |
| 2 | 1 (2) | 3 (7) | 3 (7) | 3 (7) | 2 (4) |
| 3 | 3 (7) | 2 (4) | 3 (7) | 4 (9) | 4 (9) |
| 4 | 23 (50) | 21 (46) | 20 (43) | 18 (39) | 13 (28) |
| 5 | 19 (41) | 20 (43) | 20 (43) | 21 (46) | 27 (59) |
| EPA | Minimum Level of Supervision [Median (IQR)] | Minimum Level of Supervision Based on Consensus* | ||
|---|---|---|---|---|
| Graduation† | Practice | Graduation† | Practice | |
| Manage acute complex respiratory disease | 4 (3.25–4) | 4 (4-5)‡ | 3 | 4 |
| Manage chronic respiratory disease | 4 (3–4) | 4 (4-5)‡ | 3 | 4 |
| Communicate a new diagnosis | 4 (3–4) | 4 (4-5)‡ | 3 | 4 |
| Manage respiratory equipment | 4 (3–4) | 4 (4-5)‡ | 3 | 4 |
| Perform procedures | 4 (3.25–4) | 5 (4-5)‡ | 3 | 4 |
The minimum level of supervision that PDs believed to be required for subspecialists to perform safe and effective practice are shown in Table 2. The median minimum level of supervision for practice was 4, requiring indirect supervision for complex cases, for the following four EPAs (Table 3): manage acute complex respiratory disease, manage chronic respiratory disease, communicate a new diagnosis, and manage respiratory equipment. However, for the EPA “perform procedures,” the median minimum level of supervision for safe and effective practice was 5 (unsupervised practice). However, the minimum level for practice was 4 for all EPAs, indicating that indirect supervision would still be required for complex cases.
There was a statistically significant difference between the minimum level of supervision for graduation from fellowship and safe and effective practice for each of the five EPAs (Table 3). There was no association between the minimum level of supervision for graduation or practice for any of the EPAs and the number of fellows, years as PD, or PD understanding, current use of EPAs, or involvement in the previous SPIN study (6).
The majority of pediatric pulmonary PDs reported that they would not require their fellows to be trusted to practice without supervision for graduation for any of the five pediatric pulmonology EPAs, which constitute the essential activities of our subspecialty. These findings challenge expectations described by the ACGME (10) and the Institute of Medicine (14) that trainees achieve levels of competence required for unsupervised practice. That trainees may not meet expected levels for unsupervised practice for EPAs has been reported in pediatric residency (12) and subspecialty fellowship programs (15). In the study by Schumacher and colleagues, if performance standards were set for residents to achieve unsupervised practice for graduation, they would not have been met for 9 of the 17 general pediatrics EPAs (12). In pediatric fellowship programs, PDs did not require fellows to be trusted to practice without supervision for graduation for any of the seven common pediatric subspecialty EPAs (15). Our findings suggest that the structure and curricula of training should be reevaluated to determine whether the goal of entrustment for unsupervised practice can be achieved. Models of competency-based medical education in which time is variable and outcomes are fixed should be considered (16). The use of EPAs to guide assessment and curricula would allow better alignment between training and expected outcomes and entrustment decisions to be made across the continuum from training to practice. Furthermore, structured assessment and support may be required for recent fellowship graduates during their early years of practice to ensure high-quality patient care and further skill development to ensure safe and effective outcomes (17, 18).
The reasons that the majority of pediatric pulmonary PDs reported that they would not require fellows to be trusted to practice without supervision for graduation are unclear. A potential contributing factor is that the majority of PDs did not expect subspecialists to be required to be trusted to practice without supervision to provide safe and effective care for any of the EPAs except “perform procedures.” Despite our previous work, which provided validity evidence for the use of EPAs as assessment tools in pediatric fellowship programs (6), there may be ambiguity in PD interpretation of the entrustment and supervision scales used in this study. PDs were asked to assign the level of supervision on the basis of what they would trust fellows to do, which may not align with observable activities because those can be limited by institutional supervision policies. In addition, the ACGME requirements for supervision during training may conflate the response to the survey question about minimum level of supervision needed for graduation (i.e., one cannot assess any trainee as ready for unsupervised practice because trainees are required to have some level of supervision at all times). Furthermore, the fact that the majority of PDs chose level 4 (trusted to execute with indirect supervision with a discussion of information for a few complex cases) as the minimum level of supervision for practicing subspecialists for all the EPAs except “perform procedures,” suggests that there might be some confusion about the term “supervision,” as we did not define it in the survey. Supervision in graduate medical education specifically includes observation, monitoring, and assessment of learners (with feedback) by supervisors (14); it differs from the common postgraduate practice of working unsupervised but asking for help with specific cases. We are currently collecting validity evidence for the supervision scales, including qualitative data on PD interpretation, which could help identify and clarify potential areas of confusion.
We found a significant difference between the minimum level of supervision required by pediatric pulmonary PDs for graduation and what they believed to be necessary for safe and effective practice as a subspecialist. The gap may be due to inadequate resources or expertise to adequately train fellows. Alternatively, it may reflect PD beliefs about the goals and outcomes of fellowship training and beliefs that fellows will receive adequate support to continue to develop along their trajectory of clinical expertise in their postgraduate positions. It is possible that the absence of performance data resulted in PDs setting lower minimum standards for graduation. However, previous studies indicate that, without performance data, assessors actually overestimate the minimum standards (19).
The fact that the majority of pediatric pulmonary PDs reported that they would graduate fellows even if they required supervision suggests the need for structured support during their early practice period. Supervision of new graduates may be variable in institutions and practices. However, it appears that a period of indirect supervision that includes assessment of skills and knowledge with feedback should be considered. A structured handover of PDs to the new graduate’s practice would help facilitate the transition. However, even fellows who demonstrated competency in EPAs during fellowship will be faced with challenges adapting their skills to new patient populations and healthcare systems. Ensuring that new graduates have mentors can help support quality patient care as well as their professional success and well-being (18, 20). An individualized curriculum of continuous professional development and maintenance of certification activities targeted at identified gaps can help support moving the early pulmonologists along the developmental trajectory to unsupervised practice.
Setting minimum standards for assessment tools such as EPAs is a complex process that includes the subjective judgment of assessors (21). PDs are important contributors to the credibility of the standards, as they have extensive experience assessing fellows and making summative decisions about advancement and graduation. However, when setting standards, it is critical to include other stakeholders in the decision (22), such as division directors, department chairs, trainees, graduates, institutions, patients and their families, and accrediting organizations. For this study, we set the standard for graduation and practice as the level of supervision for which no more than 20% of PDs would want a lower minimum level by consensus of SPIN members and other groups with educational experts to align with residency performance data (12). However, these PD judgments were subjective, and they need to be triangulated with fellow performance data. Qualitative data on how PDs interpret and select the minimum level of supervision would be of benefit in interpreting our study results. We are presently collecting longitudinal fellow performance and qualitative data to gain further validity evidence for the scales and proposed standards.
The limitations of the study include the potential for nonresponder and self-reporting bias. Nonresponder bias is limited, however, by the high response rate. Although this study established content and response process validity of the pediatric pulmonology EPAs and their scales, the validity argument needs to be further strengthened, including internal structure, relationship to other variables, and consequences (23). Furthermore, standard setting and consequential validity require input from other stakeholders. In addition, by adjusting the minimum level of supervision for graduation by only one level for PDs who stated that they would graduate fellows even if they did not meet the minimum level, we may have overestimated the actual level. In that case, the amount of supervision that fellows require at graduation and the gap between graduation and practice would be even greater than we reported. Although EPAs have the advantage of being construct aligned, that is, asking faculty whose responsibility it is to supervise residents in the clinical setting to assess the amount of supervision needed, further studies are needed to demonstrate the benefits of using EPAs over current methods of assessment (24).
This study suggests a need to assess in real time whether newly graduated pediatric pulmonologists are ready to practice without supervision. Depending on the results, further collaboration between PDs, program leadership, institutions, regulatory bodies, and other stakeholders may need to reevaluate the goals for fellowship training programs. Support and mentoring to ensure both safe and effective outcomes for patients and the well-being of early practice pulmonologists may also be critical. Studies collecting fellow performance data and qualitative information from PDs on their beliefs about the use of EPAs in summative assessment are presently underway.
| 1. | Carraccio C, Englander R, Gilhooly J, Mink R, Hofkosh D, Barone MA, et al. Building a Framework of Entrustable Professional Activities, Supported by Competencies and Milestones, to Bridge the Educational Continuum. Acad Med 2017;92:324–330. |
| 2. | Ten Cate O. Nuts and bolts of entrustable professional activities. J Grad Med Educ 2013;5:157–158. |
| 3. | ten Cate O, Scheele F. Competency-based postgraduate training: can we bridge the gap between theory and clinical practice? Acad Med 2007;82:542–547. |
| 4. | American Board of Pediatrics. Entrustable professional activities for subspecialties. 2021 [accessed 2021 Jan 15]. Available from: www.abp.org/subspecialty-epas.) |
| 5. | Mink RB, Myers AL, Turner DA, Carraccio CL. Competencies, milestones, and a level of supervision scale for entrustable professional activities for scholarship. Acad Med 2018;93:1668–1672. |
| 6. | Mink RB, Schwartz A, Herman BE, Turner DA, Curran ML, Myers A, et al.; Steering Committee of the Subspecialty Pediatrics Investigator Network (SPIN). Validity of level of supervision scales for assessing pediatric fellows on the common pediatric subspecialty entrustable professional activities. Acad Med 2018;93:283–291. |
| 7. | Moore D, Young CJ, Hong J. Implementing entrustable professional activities: the yellow brick road towards competency-based training? ANZ J Surg 2017;87:1001–1005. |
| 8. | Boyce P, Spratt C, Davies M, McEvoy P. Using entrustable professional activities to guide curriculum development in psychiatry training. BMC Med Educ 2011;11:96. |
| 9. | Mulder H, Ten Cate O, Daalder R, Berkvens J. Building a competency-based workplace curriculum around entrustable professional activities: the case of physician assistant training. Med Teach 2010;32:e453–e459. |
| 10. | Accreditation Council for Graduate Medical Education. ACGME Program Requirements for Graduate Medical Education. 2021 [accessed 2021 Feb 26]. Available from: https://www.acgme.org/. |
| 11. | Mink R, Schwartz A, Carraccio C, High P, Dammann C, McGann KA, et al.; Steering Committee of the Subspecialty Pediatrics Investigator Network. Creating the subspecialty pediatrics investigator network. J Pediatr 2018;192:3–4.e2. |
| 12. | Schumacher DJ, West DC, Schwartz A, Li ST, Millstein L, Griego EC, et al.; Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network General Pediatrics Entrustable Professional Activities Study Group. Longitudinal assessment of resident performance using entrustable professional activities. JAMA Netw Open 2020;3:e1919316. |
| 13. | R Core Team. The R project for statistical computing. 2021 [accessed 2020 May 8]. Available from: https://www.R-project.org.) |
| 14. | Ulmer U, Wolman DM, Johns MME. Institute of Medicine of the National Academies: resident duty hours. Enhancing sleep, supervision, and safety. Washington, DC: The National Academies Press; 2009. |
| 15. | Turner DA, Schwartz A, Carraccio C, Herman B, Weiss P, Baffa JM, et al. Continued supervision for the common pediatric subspecialty entrustable professional activities may be needed following fellowship graduation. Acad Med 2021;96:22–28. |
| 16. | Carraccio C, Englander R, Van Melle E, Ten Cate O, Lockyer J, Chan MK, et al.; International Competency-Based Medical Education Collaborators. Advancing competency-based medical education: a charter for clinician-educators. Acad Med 2016;91:645–649. |
| 17. | Librizzi J, Winer JC, Banach L, Davis A. Perceived core competency achievements of fellowship and non-fellowship-trained early career pediatric hospitalists. J Hosp Med 2015;10:373–379. |
| 18. | Westerman M, Teunissen PW, Fokkema JP, van der Vleuten CP, Scherpbier AJ, Siegert CE, et al. The transition to hospital consultant and the influence of preparedness, social support, and perception: a structural equation modelling approach. Med Teach 2013;35:320–327. |
| 19. | Kane MT, Crooks TJ, Cohen AS. Designing and evaluating standard-setting procedures for licensure and certification tests. Adv Health Sci Educ Theory Pract 1999;4:195–207. |
| 20. | Kashiwagi DT, Varkey P, Cook DA. Mentoring programs for physicians in academic medicine: a systematic review. Acad Med 2013;88:1029–1037. |
| 21. | Norcini JJ. Setting standards on educational tests. Med Educ 2003;37:464–469. |
| 22. | Yudkowsky R, Downing S, Tekian A. Standard Setting. In: Downing S, Yudkowsky R, eds. Assessment in Health Professions Education. New York, NY: Taylor and Francis; 2009:119–148. |
| 23. | Downing SM. Validity: on meaningful interpretation of assessment data. Med Educ 2003;37:830–837. |
| 24. | Crossley J, Johnson G, Booth J, Wade W. Good questions, good answers: construct alignment improves the performance of workplace-based assessment scales. Med Educ 2011;45:560–569. |
Supported in part by the American Board of Pediatrics Foundation. The content is solely the responsibility of the authors and does not necessarily represent the official views of the American Board of Pediatrics or the American Board of Pediatrics Foundation.
Author Contributions: Drs. Mink, Schwartz, Carraccio, and Herman conceptualized and designed the study, supervised the study, analyzed the data, and critically reviewed and revised the manuscript. Dr. Weiss helped to conceptualize and design the study, recruited participants, analyzed the data, drafted the initial and revised the manuscript.
This article has a data supplement, which is accessible from this issue’s table of contents at www.atsjournals.org.
Author disclosures are available with the text of this article at www.atsjournals.org.
