Abstract
Background
High stakes medical licensing programs are planning to augment and adapt current examinations to be relevant for a two-decision point model for licensure: entry into supervised practice and entry into unsupervised practice. Therefore, identifying which skills should be assessed at each decision point is critical for informing examination development, and gathering input from residency program directors is important.
Methods
Using data from previously developed surveys and expert panels, a web-delivered survey was distributed to 3,443 residency program directors. For each of the 28 procedural and 18 advanced communication skills, program directors were asked which clinical skills should be assessed, by whom, when, and how. Descriptive statistics were collected, and Intraclass Correlations (ICC) were conducted to determine consistency across different specialties.
Results
Among 347 respondents, program directors reported that all advanced communication and some procedural tasks are important to assess. The following procedures were considered ‘important’ or ‘extremely important’ to assess: sterile technique (93.8%), advanced cardiovascular life support (ACLS) (91.1%), basic life support (BLS) (90.0%), interpretation of electrocardiogram (89.4%) and blood gas (88.7%). Program directors reported that most clinical skills should be assessed at the end of the first year of residency (or later) and not before graduation from medical school. A minority were considered important to assess prior to the start of residency training: demonstration of respectfulness (64%), sterile technique (67.2%), BLS (68.9%), ACLS (65.9%) and phlebotomy (63.5%).
Discussion
Results from this study support that assessing procedural skills such as cardiac resuscitation, sterile technique, and phlebotomy would be amenable to assessment at the end of medical school, but most procedural and advanced communications skills would be amenable to assessment at the end of the first year of residency training or later.
Conclusions
Gathering data from residency program directors provides support for developing new assessment tools in high-stakes licensing examinations.
Keywords: high stakes assessment, licensing examination, procedures, communication and interpersonal skills, residency program directors
Over the last decade, the Accreditation Council on Graduate Medical Education (ACGME) and American Osteopathic Association (AOA) competencies have been integrated into graduate medical education (1, 2). As a consequence, greater attention has been given to defining competence and identifying specific skills required of residents (3). As an example, the ACGME and American Board of Pediatrics (ABP) have established the Pediatric Milestones Working Group to define specific competencies required of residents at different levels of training (4, 5). Not only are program directors attempting to clarify which skills should be assessed at which level of training, but national licensing boards are also working toward identifying clear assessment objectives (6). The two examinations used to license physicians in the United States, United States Medical Licensing Exam (USMLE) and Comprehensive Osteopathic Medical Licensing Examination – USA (COMLEX-USA) (7, 8), are phasing in a two-decision point model for licensure: entry into supervised practice and entry into unsupervised practice (9, 10). Therefore, identifying which skills should be assessed at each of these decision points becomes critical.
Preparing medical students for residency training has been the focus of medical schools in the United States (11, 12). As part of the Medical School Objectives Project (MSOP), the Association of American Medical Colleges (AAMC) has identified specific skills required of medical students prior to graduation (11), such as patient care, communication and procedural skills. Specific examples include ability to communicate effectively with patients, families and colleagues and ability to perform routine procedures such as venipuncture, lumbar puncture, laceration repair, and thoracentesis (11). In a recent study by the National Board of Medical Examiners (NBME), residents reported performing a number of procedures and communication tasks during their first few months of training (13). The authors conclude that perhaps these clinical skills should be taught and assessed prior to completion of medical school training.
Numerous studies have identified educational gaps between graduating medical students and residents, with regard to clinical skills such as advanced communication and procedural skills (14–21). For instance, Wagner and Lypson described objective standardized clinical examinations (OSCEs) administered to residents at the start of their training (14). New resident performance in communication assessments were consistently high, but patient care scores varied widely with particularly low scores in the areas of hand hygiene and aseptic technique. In another study of new residents, Lypson reported lowest OSCE scores in the areas of informed consent and identification of critical values (15). Both studies not only identified variability among entering residents, but also exposed gaps between residency program faculty expectations and actual performance reflective of undergraduate medical education training. Identifying disparate expectations of medical students and residents has challenged medical educators to identify which specific clinical skills should be taught and assessed. Previous investigators report conflicting opinions regarding which clinical skills should be taught or assessed during medical school (17–24) and residency training (18, 25, 26).
While many studies have attempted to identify the clinical skills required of trainees, most have been limited to a particular institution or discipline. Identifying these clinical skills is particularly important for developing and enhancing assessments for licensing examinations. Two high-stakes clinical skills examinations are used in the United States to assess clinical skills performance of medical students: USMLE Step 2 Clinical Skills (USMLE Step 2-CS) and COMLEX-USA Level 2-Performance Evaluation (COMLEX-USA Level 2-PE) (7, 8). However, neither clinical skills examination currently assesses advanced communication (triadic encounters, death and dying, etc), procedural, or clinical skills which may be unique to a particular specialty. To inform test development and exam enhancement, the goal of this study is to investigate residency program directors’ expectations of assessment of their residents’ procedural and advanced communication skills. In particular, the objective is to survey residency program directors and identify which clinical skills are important to assess, by whom, when, and how (formative versus summative).
Methods
Instrument (survey to residency program directors)
Using content from a variety of questionnaires used in previous studies (17–20, 22–33) and recommendations from the AAMC (11, 34, 35) and ACGME (3), physician staff from the National Board of Osteopathic Medical Examiners (NBOME) compiled a list of specific procedural skills and advanced communication skills.
Recommendations from NBOME's Clinical Skills Testing Advisory Committee (8 members) and strategic planning committee (16 members) were incorporated into the survey. Members from both committees are considered experts in medical education and assessment; members include representatives from undergraduate medical education (deans, associate deans, faculty), graduate medical education (ACGME and AOA-accredited residency program directors, directors of medical education), clinicians, medical educators, and psychometricians.
After receiving input from these expert panels, items were reviewed, and further enhancements were made by NBOME's Research Advisory Committee (12 members) composed of experts in assessment, education, and research. The final instrument (found in Supplemental content) addressed 28 procedural skills and 18 advanced communication skills, was pretested by physician staff, and distributed using Survey Monkey.
Sample
The web-delivered survey was distributed to 3,443 ACGME and AOA-accredited residency program directors with valid email addresses contained in NBOME's residency program director database. Program directors were randomly divided into two groups; each group received one of two versions of the survey (one with procedural skills presented first and another with advanced communication skills presented first). Program directors from all disciplines were included in the sample.
Analysis
Institutional Review Board approval was granted by the Center for the Advancement of Healthcare Education and Delivery (C-AHEAD) to collect, analyze and report these data for this study. Survey responses were analyzed using descriptive statistics. Intraclass Correlations (ICC), which describe the degree of group agreement, were calculated to examine the disparity in responses of program directors of different specialties.
Results
A total of 347 program directors completed the survey, representing a response rate of 10.1%. Program directors from a wide range of disciplines responded to the survey, and specialty distributions were reflective of national data (Table 1) (36, 37). For instance, 45 surgery and surgical subspecialty program directors were included in the sample (13% of the sample), compared to 946 in the national sample (19.9% of all residencies). Primary care residencies were slightly overrepresented in our sample. For instance, 100 family medicine residency program directors were included in the sample (28.8% of the sample), compared to 636 in the national sample (13.4% of all residencies). Among the 347 respondents, 44 were identified as ‘surrogates,’ program director-selected surrogates (e.g., assistant/associate residency program directors) who completed the survey on behalf of the residency program director. Among those who completed the survey, 293 respondents completed the section on procedural skills and 284 completed the section on advanced communication skills. The attrition in survey completion was attributed to the length of the survey.
Table 1.
Residency program director respondents in comparison with national sample data, by specialty (n=347)
| Sample | National | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Specialtya | ACGME-accredited residency programs n=305b | AOA-accredited residency programs n=42 | ACGME-accredited residency programs (total=4,128) (37) | AOA-accredited residency programs (total=636) (36) | |||||
| Family Med | 84 | 27.5% | 16 | 38.1% | 452 | 10.9% | 184 | 28.9% | |
| Internal Med | 40 | 13.1% | 11 | 26.2% | 379 | 9.2% | 88 | 13.8% | |
| Other | 27 | 8.8% | 2 | 4.8% | 1168 | 28.3% | 109 | 17.1% | |
| Surgical Subsp | 26 | 8.5% | 4 | 9.5% | 581 | 14.1% | 78 | 12.3% | |
| Pediatrics | 25 | 8.2% | 1 | 2.4% | 196 | 4.7% | 17 | 2.7% | |
| OB/GYN | 22 | 7.2% | 2 | 4.8% | 246 | 6.0% | 31 | 4.9% | |
| Emergency Med | 16 | 5.2% | 10 | 23.8% | 153 | 3.7% | 43 | 6.8% | |
| Psychiatry | 15 | 4.9% | – | – | 182 | 4.4% | 9 | 1.4% | |
| Anesthesiology | 13 | 4.3% | – | – | 132 | 3.2% | 12 | 1.9% | |
| Radiology | 13 | 4.3% | – | – | 188 | 4.6% | 14 | 2.2% | |
| PM & R | 11 | 3.6% | – | – | 79 | 1.9% | 3 | 0.5% | |
| Surgery | 11 | 3.6% | 4 | 9.5% | 246 | 6.0% | 41 | 6.4% | |
| Neurology | 9 | 2.9% | 1 | 2.4% | 126 | 3.1% | 7 | 1.1% | |
Abbreviations: ACGME, Accreditation Council on Graduate Medical Education; AOA, American Osteopathic Association; ObGYN, Obstetrics and Gynecology; PM & R, Physical Medicine and Rehabilitation.
Eleven program directors reported to have more than one specialty so that each of the total percentages corresponding to sample subgroups may exceed 1.
Sixty-four of the ACGME-accredited residency programs are also certified by the AOA.
Procedural skills
Figure 1 presents program directors’ opinions about the importance of assessing 28 procedural skills. Program directors considered a number of procedures to be important to assess (sum of ‘important’ and ‘extremely important’ responses): sterile technique (93.8%), advanced cardiovascular life support (ACLS) (91.1%), basic life support (BLS) (90.0%), interpretation of EKG (89.4%), and interpretation of blood gas (88.7%). Skills such as osteopathic manipulative treatment (35.6%), obtaining a blood culture (37.0%), and PPD placement (38.4%) were considered less important.
Fig. 1.
Importance of procedural skills assessment by program directors’ responses. The values reflect responses to the survey question ‘In your opinion, how important is it for each of the following skills to be assessed?’ Values reflect the sum of responses to ‘important’ and ‘extremely important’. The skills are sorted in the descending order of importance rated by all program directors (n=293).
With regard to the importance of assessing procedures, agreement varied among program directors of different specialties. ICC is an index representing proportion of the total variance explained by group effects with higher ICC values indicating larger group variation (lower agreement between specialty groups). ICC values ranged from 0.04 to 0.51. Signifying disagreement between program directors from different specialties, high levels of disparity between specialty groups (ICC values >0.30) were found for central line access, lumbar puncture, incision and drainage, splinting/casting, child birth (vaginal) and pelvic exam. Signifying agreement between program directors from different specialties, low levels of disparity (ICC values <0.10) were found for nasogastric tube placement, obtaining blood culture, cardiac resuscitation (BLS), phlebotomy, sterile technique, and injection (IM/SC). Some procedural skills, such as sterile technique and cardiac resuscitation (BLS), displayed both low group disparity and high importance ratings.
Presented in Fig. 2 are program directors’ opinions regarding how the 28 procedural skills should be assessed. Program directors overwhelmingly reported that each of the procedures should be assessed in a formative fashion, followed by a combination of both formative and summative assessment. Compared to other procedures, ACLS (24.2%), BLS (23.6%), Neonatal Advanced Life Support (NALS) (21.3%), Pediatric Advanced Life Support (PALS) (20.7%), phlebotomy (15.2%), sterile technique (14.9%), injection (14.7%) and intravenous placement (14.7%) were considered to be procedures amenable to summative assessment.
Fig. 2.
Type of assessment for procedural skills by program directors responses. The values reflect responses to the survey question ‘In your opinion, please mark whether the assessments should be summative (e.g., used for advancement purposes), formative (e.g., used for feedback and teaching purposes), both or neither.’ Procedural skills are presented in the descending order of responses to ‘Summative Assessment’ (%) by program directors (n=293).
The majority of program directors reported residency program faculty to be the most appropriate for assessing procedural skills (Table 2). Only for the phlebotomy skill, medical school faculty were regarded more appropriate than residency program faculty. A small number of program directors reported that resuscitation (ACLS, BLS, PALS and NALS) could be evaluated in a high-stakes testing environment (28.7, 27.6, 27.6 and 27.0%, respectively).
Table 2.
Program directors’ perception of who should be evaluating procedural skillsa
| Procedural skills | Medical School Faculty (%) | Residency Program Faculty (%) | Residency Program Director (%) | Director of Graduate Medical Education (%) | High Stakes Testing/Licensing Exam Agency (%) |
|---|---|---|---|---|---|
| Sterile technique | 49.5 | 69.3 | 28.0 | 5.5 | 6.5 |
| Cardiac resuscitation (ACLS) | 30.4 | 52.6 | 28.0 | 8.2 | 28.7 |
| Cardiac resuscitation (BLS) | 32.8 | 49.5 | 27.0 | 7.2 | 27.6 |
| Interpretation of EKG | 42.7 | 71.7 | 28.7 | 3.4 | 7.9 |
| Interpretation of arterial blood gas | 44.0 | 70.3 | 28.7 | 4.1 | 7.2 |
| Suturing | 41.6 | 76.8 | 27.3 | 3.1 | 1.7 |
| Pelvic exam | 47.1 | 69.3 | 26.3 | 4.4 | 4.1 |
| Incision and drainage (wound/abscess) | 23.9 | 80.9 | 27.0 | 4.1 | 1.7 |
| Injection (IM/SC) | 42.0 | 63.1 | 22.9 | 3.1 | 2.4 |
| Lumbar puncture | 25.6 | 81.6 | 28.3 | 4.1 | 1.4 |
| Cardiac resuscitation (PALS) | 27.6 | 50.9 | 26.3 | 5.8 | 27.6 |
| Endotracheal tube insertion | 21.5 | 77.8 | 29.7 | 4.1 | 3.1 |
| Cardiac resuscitation (NALS) | 26.6 | 49.5 | 25.6 | 5.5 | 27.0 |
| Urinary catheter placement | 42.3 | 64.5 | 21.8 | 3.1 | 2.0 |
| Nasogastric tube placement | 37.2 | 70.0 | 23.5 | 3.4 | 1.4 |
| Intravenous catheter placement | 46.1 | 59.7 | 22.9 | 3.4 | 3.1 |
| Central line access | 19.1 | 77.1 | 26.6 | 3.8 | 3.1 |
| Phlebotomy | 54.6 | 53.9 | 21.8 | 2.7 | 3.4 |
| Splinting/casting | 33.8 | 73.0 | 26.6 | 4.1 | 2.4 |
| Central line placement | 18.4 | 80.2 | 28.7 | 4.1 | 2.7 |
| Child birth (vaginal) | 28.7 | 72.7 | 23.9 | 4.4 | 1.7 |
| Spirometry | 36.9 | 67.9 | 24.9 | 2.7 | 3.8 |
| Arterial puncture | 37.5 | 68.9 | 25.6 | 3.4 | 2.4 |
| Thoracentesis | 22.2 | 78.2 | 25.9 | 3.1 | 1.0 |
| Paracentesis | 22.5 | 77.8 | 26.3 | 3.4 | 1.4 |
| PPD placement | 48.1 | 54.6 | 20.8 | 2.4 | 3.1 |
| Blood culture | 45.1 | 55.3 | 23.2 | 3.1 | 2.4 |
| Osteopathic Manipulative Treatment (OMT) | 45.7 | 59.4 | 24.2% | 6.5 | 6.8 |
The values reflect responses to the survey question ‘In your opinion, WHO would be the most appropriate to make such judgments?’ The skills are sorted in the descending order of importance rated by all program directors (n=293) (Fig. 1).
As for the most appropriate time to assess the procedural skills (Table 3), program directors reported that assessment of most procedures should be completed at the end of the first year of residency or later. Of the responses for ‘end of the first year of residency,’ the largest rates were reported for suturing (62.8%), lumbar puncture (61.1%), and incision and drainage (60.8%). A small number of skills were considered important to assess prior to the start of residency: BLS (68.9%), sterile technique (67.2%), ACLS (65.9%), and phlebotomy (63.5%).
Table 3.
Program directors’ perception of when procedural skills should be assesseda
| Procedural skills | Prior to start of residency (%) | End of first year of residency (%) | End of second year of residency (%) | Toward the end of residency (%) | After completion of residency (%) |
|---|---|---|---|---|---|
| Sterile technique | 67.2 | 42.3 | 9.2 | 10.2 | 5.1 |
| Cardiac resuscitation (ACLS) | 65.9 | 34.8 | 12.3 | 11.3 | 5.5 |
| Cardiac resuscitation (BLS) | 68.9 | 30.4 | 10.6 | 10.9 | 5.1 |
| Interpretation of EKG | 47.4 | 56.7 | 17.7 | 12.3 | 3.4 |
| Interpretation of arterial blood gas | 49.5 | 55.3 | 12.6 | 8.9 | 3.1 |
| Suturing | 42.0 | 62.8 | 15.4 | 11.9 | 4.8 |
| Pelvic exam | 56.3 | 50.5 | 11.3 | 7.5 | 3.1 |
| Incision and drainage (wound/abscess) | 20.1 | 60.8 | 21.8 | 13.3 | 3.8 |
| Injection (IM/SC) | 50.5 | 42.3 | 9.6 | 8.5 | 3.1 |
| Lumbar puncture | 18.4 | 61.1 | 25.6 | 14.3 | 2.7 |
| Cardiac resuscitation (PALS) | 49.1 | 40.3 | 10.6 | 11.9 | 5.5 |
| Endotracheal tube insertion | 19.1 | 57.7 | 25.6 | 14.7 | 3.1 |
| Cardiac resuscitation (NALS) | 48.8 | 39.9 | 10.2 | 11.9 | 5.1 |
| Urinary catheter placement | 50.2 | 48.8 | 7.5 | 6.1 | 2.7 |
| Nasogastric tube placement | 38.2 | 56.0 | 9.2 | 7.9 | 2.0 |
| Intravenous catheter placement | 51.5 | 45.7 | 7.2 | 7.2 | 2.0 |
| Central line access | 13.7 | 58.0 | 24.9 | 14.7 | 3.8 |
| Phlebotomy | 63.5 | 35.8 | 4.8 | 6.1 | 2.0 |
| Splinting/casting | 28.7 | 50.5 | 23.5 | 17.1 | 4.1 |
| Central line placement | 11.9 | 56.3 | 27.3 | 16.0 | 4.1 |
| Child birth (vaginal) | 22.2 | 49.8 | 20.8 | 18.1 | 4.8 |
| Spirometry | 36.9 | 46.1 | 17.7 | 14.3 | 3.4 |
| Arterial puncture | 39.2 | 51.9 | 10.2 | 8.5 | 1.7 |
| Thoracentesis | 14.7 | 46.8 | 30.7 | 19.1 | 5.5 |
| Paracentesis | 17.1 | 43.7 | 30.4 | 20.5 | 4.8 |
| PPD placement | 57.3 | 36.5 | 5.5 | 5.8 | 2.0 |
| Blood culture | 53.2 | 39.9 | 4.8 | 5.8 | 1.7 |
| Osteopathic Manipulative Treatment (OMT) | 50.2 | 31.1 | 20.1 | 16.7 | 8.2 |
The values reflect responses to the survey question ‘In your opinion, WHEN would be the most appropriate to make such judgments?’ The skills are sorted in the descending order of importance rated by all program directors (n=293) (Fig. 1).
Advanced communication and interpersonal skills
Figure 3 displays program directors’ opinions about the importance of assessing 18 advanced communication skills. Program directors considered most communication skills important to assess (sum of ‘important’ and ‘extremely important’ responses). Responses were the highest for demonstrating professionalism (99.6%), respectfulness (98.9%), good listening skills (98.6%), communication with nursing/ancillary staff (98.6%), and empathy (97.9%). The remaining skills each received ratings of importance higher than 78%. The ICC coefficients examining group agreement between program directors of different specialties ranged from near 0 to 0.13. No significant between-group variation was found.
Fig. 3.
Importance of advanced communication skills assessment by program directors’ responses. The values reflect responses to the survey question ‘In your opinion, how important is it for each of the following skills to be assessed?’ Values reflect the sum of responses to ‘important’ and ‘extremely important.’ The skills are sorted in the descending order of importance rated by all program directors (n=284).
Regarding the 18 communication skills, program directors overwhelmingly reported that integrative evaluations using both summative and formative assessment should be utilized (Fig. 4). Exclusive summative assessment was not considered a suitable format by most program directors. The majority of the program directors reported residency program faculty to be the most appropriate to assess advanced communication skills (Table 4).
Fig. 4.
Type of assessment for advanced communication skills by program directors’ responses. The values reflect responses to the survey question ‘In your opinion, please mark whether the assessments should be summative (e.g., used for advancement purposes), formative (e.g., used for feedback and teaching purposes), both or neither.’ Advanced communication skills are presented in the descending order of responses to ‘Summative Assessment’ (%) by program directors (n=284).
Table 4.
Program directors’ perception of who should be evaluating advanced communication skillsa
| Communication skills | Medical School Faculty (%) | Residency Program Faculty (%) | Residency Program Director (%) | Director of Graduate Medical Education (%) | High Stakes Testing/Licensing Exam Agency (%) |
|---|---|---|---|---|---|
| Demonstrating professionalism | 47.5 | 77.1 | 54.9 | 15.8 | 8.8 |
| Demonstrating respectfulness | 51.4 | 77.8 | 47.5 | 11.3 | 6.0 |
| Demonstrating good listening skills | 52.5 | 77.5 | 44.0 | 5.6 | 6.0 |
| Communication with nursing/ancillary staff (oral) | 39.4 | 80.6 | 42.6 | 7.0 | 2.1 |
| Demonstrating empathy | 49.6 | 76.8 | 41.5 | 5.6 | 3.9 |
| Eliciting information | 52.5 | 79.9 | 39.4 | 6.3 | 7.7 |
| Giving information | 48.2 | 81.3 | 40.8 | 6.7 | 6.7 |
| Acknowledgment of medical error/mistake | 41.2 | 77.1 | 47.5 | 10.2 | 5.3 |
| Hand offs (e.g., sign out rounds, transfer of care) | 34.2 | 82.4 | 45.1 | 8.5 | 4.2 |
| Delivering bad news | 37.3 | 80.6 | 39.1 | 5.6 | 4.2 |
| Obtaining informed consent | 34.5 | 85.2 | 40.5 | 5.3 | 6.3 |
| Communication with nursing/ancillary staff (written) | 38.0 | 80.6 | 40.8 | 7.0 | 1.8 |
| Referral to consultants (oral) | 32.0 | 85.9 | 37.7 | 4.9 | 2.1 |
| Referral to consultants (written) | 32.4 | 84.9 | 37.7 | 4.6 | 1.8 |
| Demonstrating cultural competence | 47.5 | 72.5 | 43.0 | 10.6 | 6.3 |
| End of life (e.g., advance directives) | 36.3 | 79.6 | 38.7 | 7.0 | 4.6 |
| Triadic encounters (e.g., communication with parent and child) | 39.4 | 81.7 | 35.6 | 3.9 | 2.8 |
| Dictation of medical record | 31.7 | 79.9 | 41.9 | 8.5 | 2.1 |
The values reflect responses to the survey question ‘In your opinion, WHO would be the most appropriate to make such judgments?’ The skills are sorted in the descending order of importance rated by all program directors (n=284) (Fig. 3).
For all communication skills, except ‘demonstrating respectfulness’, program directors reported the end of first year of residency to be the most appropriate time for evaluation (Table 5). Of the responses for ‘end of the first year of residency,’ the largest rates were for handoffs (83.1%), referral to consultants-oral (76.4%) and dictation of medical record (77.5%).
Table 5.
Program directors’ perception of when advanced communication skills should be assesseda
| Communication skills | Prior to start of residency (%) | End of first year of residency (%) | End of second year of residency (%) | Toward the end of residency (%) | After completion of residency (%) |
|---|---|---|---|---|---|
| Demonstrating professionalism | 60.2 | 64.1 | 35.9 | 33.5 | 20.1 |
| Demonstrating respectfulness | 64.1 | 60.9 | 32.0 | 31.3 | 15.5 |
| Demonstrating good listening skills | 60.6 | 63.7 | 32.0 | 27.1 | 11.6 |
| Communication with nursing/ancillary staff (oral) | 43.3 | 73.6 | 27.8 | 26.1 | 10.9 |
| Demonstrating empathy | 57.0 | 62.7 | 32.4 | 30.3 | 12.3 |
| Eliciting information | 53.2 | 66.5 | 30.3 | 25.4 | 10.2 |
| Giving information | 42.3 | 69.0 | 37.0 | 29.2 | 11.6 |
| Acknowledgment of medical error/mistake | 39.8 | 65.1 | 34.2 | 29.2 | 13.4 |
| Hand offs (e.g., sign out rounds, transfer of care) | 27.8 | 83.1 | 35.2 | 25.7 | 9.5 |
| Delivering bad news | 25.7 | 62.0 | 41.9 | 28.5 | 10.6 |
| Obtaining informed consent | 23.2 | 77.1 | 39.8 | 26.8 | 11.3 |
| Communication with nursing/ancillary staff (written) | 37.7 | 75.0 | 28.5 | 24.6 | 10.6 |
| Referral to consultants (oral) | 16.5 | 76.4 | 37.3 | 27.5 | 9.2 |
| Referral to consultants (written) | 16.5 | 75.7 | 37.0 | 28.5 | 9.5 |
| Demonstrating cultural competence | 48.2 | 57.7 | 32.7 | 28.2 | 9.9 |
| End of life (e.g., advance directives) | 23.6 | 62.0 | 40.8 | 33.5 | 11.6 |
| Triadic encounters (e.g., communication with parent and child) | 25.0 | 69.7 | 34.5 | 27.5 | 7.7 |
| Dictation of medical record | 20.1 | 77.5 | 29.6 | 26.1 | 8.8 |
The values reflect responses to the survey question ‘In your opinion, WHEN would be the most appropriate to make such judgments?’ The skills are sorted in the descending order of importance rated by all program directors (n=284) (Fig. 3).
Discussion
Program directors reported that all advanced communication tasks and some procedural tasks are important to assess during medical training. Although their responses were consistent across disciplines when considering communication tasks, there was variability among groups when asked about procedures. High levels of agreement between program directors of different specialties were seen for nasogastric tube placement, obtaining a blood culture, cardiac resuscitation, phlebotomy, sterile technique and injections. Strong agreement is likely explained by the fact that these procedures are common to all physicians, not just those of a particular discipline. Identifying consistency among program directors of different disciplines is important, given the recent growth of specialization in graduate medical education (38–40). However, of these procedures with high levels of agreement, only cardiac resuscitation, sterile technique and injection were considered important to assess. In a similar survey of program directors, 89.7% expected competency three months into residency with regard to BLS, and 74.4% with regard to ACLS (18).
For both advanced communication and procedural skills, program directors reported that assessments should include a combination of formative and summative evaluation. This was particularly true for advanced communication tasks, demonstrated by a small number of program directors advocating for exclusive summative assessment. Compared to the other procedural skills, cardiac resuscitation, phlebotomy, sterile technique, injection and intravenous placement were considered amenable to summative assessment.
Program directors reported that most clinical skills should be assessed at the end of the first year of residency (or later) and not before graduation from medical school. Exceptions to this include demonstration of respectfulness, sterile technique, cardiac resuscitation, and phlebotomy; these were considered important to assess before the start of residency. This is a departure from the recommended procedures specified in AAMC's MSOP report, which advocates that students demonstrate the ability to complete the following eight procedures: venipuncture, inserting an intravenous catheter, arterial puncture, thoracentesis, lumbar puncture, inserting a nasogastric tube, inserting a foley catheter, and suturing lacerations (11). Among this list from the MSOP, only venipuncture (or phlebotomy) was considered important to assess at the end of medical school in our study, and the remainder were considered important to assess during the first year of residency or later. Consistent with Raymond's findings that few residents report performing specific procedures early in residency (13), our study of program directors supports that most clinical procedures should be assessed at the end of first year of residency (or later). Similarly, many of the clinical skills tasks assessed by the Medical Council of Canada Qualifying Examination Part II (MCCQE Part II) necessitate clinical experience during residency, and therefore examinees are required to complete a minimum of 12 months of postgraduate training before taking the clinical skills exam (7).
This study has a few notable limitations. First, although the sample of program directors includes the largest sample of physicians from different institutions and disciplines than any other study we could locate addressing communication and procedural skills (347 program directors), the survey response rate was 10.1%, and a higher response rate may provide additional information. Second, the program directors’ rationale for their responses was not elicited, and future study could be improved by complementing the survey with focus group discussion. Third, we did not solicit responses from medical school faculty (e.g., clerkship directors). Perspectives of medical school faculty is important to incorporate in future study, particularly since significant differences of opinion have been reported regarding which skills should be taught in medical school (17–24) and residency training (18, 25, 26). Fourth, the study did not include a formal resident task analysis with verification of completion of procedures; obtaining primary verification (such as a review of credentialing logs) would provide valuable information.
Conclusions
Performing clinical skills in a competent fashion is important for patient care. Ideally, assessments used for licensure should measure clinical skills considered important to assess among residency program directors across all disciplines and amenable to summative high-stakes assessment. As USMLE and COMLEX-USA examination programs begin to augment and adapt current examinations to comply with a two decision point model for licensure, clarifying which skills should be assessed at specific levels of training (entry into supervised practice and entry into unsupervised practice) becomes particularly important. Results from this study support that assessing procedural skills such as cardiac resuscitation, sterile technique, and phlebotomy would be important to assess at the end of medical school (entry into supervised practice), but that the assessment of most procedural and advanced communications skills would be more suited at the end of the first year of residency training or later (entry into unsupervised practice). Gathering data from residency program directors provides support for examination development as new assessment tools are considered for high-stakes licensing examinations.
Acknowledgements
We wish to thank the 347 residency program directors for sharing their opinions used in this study and Kristie Lang (NBOME) for her critical review of the manuscript.
To access the supplementary material to this article please see Supplementary Files under Article Tools online.
Conflict of interest and funding
The authors have not received any funding or benefits from industry or elsewhere to conduct this study.
References
- 1.Accreditation Council on Graduate Medical Education. Common program requirements. 2011. Available from: http://www.acgme.org/acWebsite/dutyHours/dh_dutyhoursCommonPR07012007.pdf [cited 16 May 2012]
- 2.American Osteopathic Association. Core Competency Compliance Program (CCCP) Part III. 2004. Available from: http://www.do-online.org/pdf/acc_cccppart3.pdf [cited 16 May 2012]
- 3.Swing SR. The ACGME outcome project: retrospective and prospective. Med Teach. 2007;29:648–54. doi: 10.1080/01421590701392903. [DOI] [PubMed] [Google Scholar]
- 4.Hicks PJ, Englander R, Schumacher DG, Burke A, Benson BJ, Guralnick S, et al. Pediatrics milestone project: next steps toward meaningful outcomes assessment. J Grad Med Educ. 2010;2:577–84. doi: 10.4300/JGME-D-10-00157.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Carraccio C, Burke AE. Beyond competencies and milestones: adding meaning through context. J Grad Med Educ. 2010;2:419–22. doi: 10.4300/JGME-D-10-00127.1. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.National Board of Osteopathic Medical Examiners. Fundamental osteopathic medical competency domains document. Available from: http://www.nbome.org/docs/NBOME%20Fundamental%20Osteopathic%20Medical%20Competencies.pdf [cited 16 May 2012]
- 7.Boulet J, Smee SM, Dillon GF, Gimpel JR. The use of standardized patient assessments for certification and licensure decisions. Sim Healthc. 2010;4:35–42. doi: 10.1097/SIH.0b013e318182fc6c. [DOI] [PubMed] [Google Scholar]
- 8.Langenau EE, Dyer C, Roberts WL, Wilson C, Gimpel J. Five-year summary of COMLEX-USA level 2-PE examinee performance and survey data. J Am Osteopath Assoc. 2010;110:114–25. [PubMed] [Google Scholar]
- 9.National Board of Medical Examiners. Advisory Committee For Medical School Programs. 2010. Available from: https://www.aamc.org/download/182096/data/nbme_advisory_committee_june_2010.pdf [cited 11 June 2012]
- 10.National Board of Osteopathic Medical Examiners. Enhancing COMLEX-USA. Report by the Blue Ribbon Panel of the National Board of Osteopathic Medical Examiners. 2012. Available from: http://www.nbome.org/docs/BRP-Enhancing%20COMLEX-USA%20final%20March%202012.pdf [cited 9 June 2012]
- 11.Association of American Medical Colleges (AAMC) Medical School Objectives Project (MSOP) Writing Group. Learning objectives for medical student education – guidelines for medical schools: report I of the Medical School Objectives Project. Acad Med. 1999;74:13–8. doi: 10.1097/00001888-199901000-00010. [DOI] [PubMed] [Google Scholar]
- 12.Shannon SC, Teitelbaum HS. The status and future of osteopathic medical education in the United States. Acad Med. 2009;84:707–11. doi: 10.1097/ACM.0b013e3181a43be8. [DOI] [PubMed] [Google Scholar]
- 13.Raymond MR, Mee J, King A, Haist SA, Winward ML. What new residents do during their initial months of training. Acad Med. 2011;86(Suppl 10):S59–62. doi: 10.1097/ACM.0b013e31822a70ff. [DOI] [PubMed] [Google Scholar]
- 14.Wagner D, Lypson ML. Centralized assessment in graduate medical education: cents and sensibilities. J Grad Med Educ. 2009;1:21–7. doi: 10.4300/01.01.0004. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Lypson ML, Frohna JG, Gruppen LD, Woolliscroft JO. Assessing residents’ competencies at baseline: identifying the gaps. Acad Med. 2004;79:564–70. doi: 10.1097/00001888-200406000-00013. [DOI] [PubMed] [Google Scholar]
- 16.Preston P. Commentary: centralized assessment in graduate medical education: how can it help us reinvent training? J Grad Med Educ. 2009;1:28–9. doi: 10.4300/01.01.0005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Fitch MT, Kearns S, Manthey DE. Faculty physicians and new physicians disagree about which procedures are essential to learn in medical school. Med Teach. 2009;31:342–7. doi: 10.1080/01421590802520964. [DOI] [PubMed] [Google Scholar]
- 18.Langdale LA, Schaad D, Wipf J, Marshall S, Vontver L, Scott CS. Preparing graduates for the first year of residency: are medical schools meeting the need? Acad Med. 2003;78:39–44. doi: 10.1097/00001888-200301000-00009. [DOI] [PubMed] [Google Scholar]
- 19.Lyss-Lerman P, Teherani A, Aagaard E, Loeser H, Cooke M, Harper GM. What training is needed in the fourth year of medical school? Views of residency program directors. Acad Med. 2009;84:823–9. doi: 10.1097/ACM.0b013e3181a82426. [DOI] [PubMed] [Google Scholar]
- 20.Turner SR, Hanson J, de Gara CJ. Procedural skills: what's taught in medical school, what ought to be? Educ Health (Abingdon) 2007;20:9. [PubMed] [Google Scholar]
- 21.Turner SR, de Gara CJ. Medical students and recent graduates may disagree on the importance of procedural skills education. Med Teach. 2010;32:182. [PubMed] [Google Scholar]
- 22.Elnicki DM, van Londen J, Hemmer PA, Fagan M, Wong R. US and Canadian internal medicine clerkship directors’ opinions about teaching procedural and interpretive skills to medical students. Acad Med. 2004;79:1108–13. doi: 10.1097/00001888-200411000-00022. [DOI] [PubMed] [Google Scholar]
- 23.Kowlowitz V, Curtis P, Sloane PD. The procedural skills of medical students: expectations and experiences. Acad Med. 1990;65:656–58. doi: 10.1097/00001888-199010000-00016. [DOI] [PubMed] [Google Scholar]
- 24.Magarian GJ, Mazur DJ. The procedural and interpretive skills that third-year medicine clerks should master: views of medicine clerkship directors. J Gen Intern Med. 1991;6:469–71. doi: 10.1007/BF02598175. [DOI] [PubMed] [Google Scholar]
- 25.Norris TE, Cullison SW, Fihn SD. Teaching procedural skills. J Gen Intern Med. 1997;12(Suppl 2):S64–70. doi: 10.1046/j.1525-1497.12.s2.9.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 26.Tenore JL, Sharp LK, Lipsky MS. A national survey of procedural skill requirements in family practice residency programs. Fam Med. 2001;33:28–38. [PubMed] [Google Scholar]
- 27.Wu EH, Elnicki DM, Alper EJ, Bost JE, Corbett EC, Jr, Fagan MJ, et al. Procedural and interpretive skills of medical students: experiences and attitudes of fourth-year students. Acad Med. 2008;83(Suppl 10):S63–67. doi: 10.1097/ACM.0b013e318183c5a7. [DOI] [PubMed] [Google Scholar]
- 28.Sanders CW, Edwards JC, Burdenski TK. A survey of basic technical skills of medical students. Acad Med. 2004;79:873–5. doi: 10.1097/00001888-200409000-00013. [DOI] [PubMed] [Google Scholar]
- 29.Wigton RS, Blank LL, Nicolas JA, Tape TG. Procedural skills training in internal medicine residencies. A survey of program directors. Ann Intern Med. 1989;111:932–8. doi: 10.7326/0003-4819-111-11-932. [DOI] [PubMed] [Google Scholar]
- 30.Wickstrom GC, Kolar MM, Keyserling TC, Kelley DK, Xie SX, Bognar BA, et al. Confidence of graduating internal medicine residents to perform ambulatory procedures. J Gen Intern Med. 2000;15:361–5. doi: 10.1046/j.1525-1497.2000.04118.x. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 31.Hobgood CD, Riviello RJ, Jouriles N, Hamilton G. Assessment of communication and interpersonal skills competencies. Acad Emerg Med. 2002;9:1257–69. doi: 10.1111/j.1553-2712.2002.tb01586.x. [DOI] [PubMed] [Google Scholar]
- 32.Makoul G. Essential elements of communication in medical encounters: the Kalamazoo consensus statement. Acad Med. 2001;76:390–3. doi: 10.1097/00001888-200104000-00021. [DOI] [PubMed] [Google Scholar]
- 33.Duffy FD, Gordon GH, Whelan G, Cole-Kelly K, Frankel R, Buffone N, et al. Assessing competence in communication and interpersonal skills: the Kalamazoo II report. Acad Med. 2004;79:495–507. doi: 10.1097/00001888-200406000-00002. [DOI] [PubMed] [Google Scholar]
- 34.Association of American Medical Colleges. Recommendation for clinical skills curricula for undergraduate medical education. 2005. Available from: https://members.aamc.org/eweb/upload/Recommendations%20for%20Clinical%20Skills%20Curricula%202005.pdf [cited 16 May 2012]
- 35.Association of American Medical Colleges. Recommendations for Preclerkship Clinical Skills Education for Undergraduate Medical Education. 2008. Available from: https://www.aamc.org/download/163788/data/recommendations_for_preclerkship_skills_education_for_ugme.pdf [cited 16 May 2012]
- 36.Freeman E, Duffy T, Lischka TA. Osteopathic graduate medical education 2010. J Am Osteopath Assoc. 2010;110:150–9. [PubMed] [Google Scholar]
- 37.Brotherton SE, Etzel SI. Graduate medical education, 2009–2010. JAMA. 2010;304:1255–70. doi: 10.1001/jama.2010.1273. [DOI] [PubMed] [Google Scholar]
- 38.Barondess JA. Specialization and the physician workforce: drivers and determinants. JAMA. 2000;284:1299–301. doi: 10.1001/jama.284.10.1299. [DOI] [PubMed] [Google Scholar]
- 39.Donini-Lenhoff FG, Hedrick HL. Growth of specialization in graduate medical education. JAMA. 2000;284:1284–9. doi: 10.1001/jama.284.10.1284. [DOI] [PubMed] [Google Scholar]
- 40.Julian K, Riegels NS, Baron RB. Perspective: creating the next generation of general internists: a call for medical education reform. Acad Med. 2011;86:1443–7. doi: 10.1097/ACM.0b013e3182303a32. [DOI] [PubMed] [Google Scholar]