Association of Volume of Patient Encounters with Residents’ In-Training Examination Performance

ABSTRACT

BACKGROUND

Patient care and medical knowledge are Accreditation Council for Graduate Medical Education (ACGME) core competencies. The correlation between amount of patient contact and knowledge acquisition is not known.

OBJECTIVE

To determine if a correlation exists between the number of patient encounters and in-training exam (ITE) scores in internal medicine (IM) and pediatric residents at a large academic medical center.

DESIGN

Retrospective cohort study

PARTICIPANTS

Resident physicians at Mayo Clinic from July 2006 to June 2010 in IM (318 resident-years) and pediatrics (66 resident-years).

METHODS

We tabulated patient encounters through review of clinical notes in an electronic medical record during post graduate year (PGY)-1 and PGY-2. Using linear regression models, we investigated associations between ITE score and number of notes during the previous PGY, adjusted for previous ITE score, gender, medical school origin, and conference attendance.

KEY RESULTS

For IM, PGY-2 admission and consult encounters in the hospital and specialty clinics had a positive linear association with ITE-3 % score (β = 0.02; p = 0.004). For IM, PGY-1 conference attendance is positively associated with PGY-2 ITE performance. We did not detect a correlation between PGY-1 patient encounters and subsequent ITE scores for IM or pediatric residents. No association was found between IM PGY-2 ITE score and inpatient, outpatient, or total encounters in the first year of training. Resident continuity clinic and total encounters were not associated with change in PGY-3 ITE score.

CONCLUSIONS

We identified a positive association between hospital and subspecialty encounters during the second year of IM training and subsequent ITE score, such that each additional 50 encounters were associated with an increase of 1 % correct in PGY-3 ITE score after controlling for previous ITE performance and continuity clinic encounters. We did not find a correlation for volume of encounters and medical knowledge for IM PGY-1 residents or the pediatric cohort.

INTRODUCTION

Patient care and medical knowledge are Accreditation Council for Graduate Medical Education (ACGME) core competencies for all resident physicians. However, it is not known how much knowledge is derived from the direct care of patients. Recent changes to duty hour requirements¹ have the potential effect of reducing patient clinical encounters, yet the effect of the volume of clinical encounters on resident education is not known. The In-Training Examination (ITE) is a standardized test that is used to assess resident knowledge,² and has been shown to correlate with American Board of Internal Medicine (ABIM)³ and American Board of Pediatrics General Pediatrics⁴ certification examination success. Previous research has correlated independent reading and conference attendance to ITE score.^5,6 Residents report reading the medical literature in the context of patient care as being most useful to their learning.⁷ Research involving medical students during their internal medicine (IM) clerkship has shown a weak correlation between student-reported volume of encounters and end of rotation examinations,⁸ though other studies have found no correlation. ^9–12

We sought to determine if there is a correlation between the volume of patient encounters and the change in year-to-year ITE scores for IM and pediatric residents at a large academic hospital. To identify encounters that would provide the highest yield for medical knowledge, we selected encounters with patients presenting with undifferentiated problems requiring assessment by a particular resident. This includes admission to the hospital, admission to the intensive care unit, and presentation to the outpatient clinic. We hypothesized that the association between patient encounters and knowledge could take one of several forms (Fig. 1):

1. Linear relationship: in a positive linear relationship, residents with more encounters demonstrate greater medical knowledge, with each additional encounter correlating with a consistent gain in medical knowledge.

2. Threshold: a linear relationship until a threshold is reached, above which point no additional knowledge is gained or lost.

3. Yerkes Dodson curve:¹³ a positive association up to a critical point, after which there is a negative association.

4. Null (not shown): no association between patient encounters and ITE scores.

Figure 1.

Models for knowledge acquisition versus volume of patient encounters. Positive linear association (solid): each additional encounter provides educational value to the learner. Threshold (dashed): a linear relationship until a threshold is reached, above which point no additional knowledge is gained from each subsequent encounter. Yerkes Dodson curve (gray): increased knowledge acquisition up to a critical point; residents beyond the peak do not have time to synthesis the educational opportunity or are otherwise stressed by the workload such that their educational performance is poorer when compared to their colleagues with fewer encounters. Null (not shown): no association between patient encounters and exam scores.

In the linear model with a positive association, reductions in the number of patient encounters would result in less medical knowledge, whereas in a threshold model, reductions in clinical encounters would not affect medical knowledge as long as the number of encounters remained above the threshold. A Yerkes-Dodson curve would suggest that residents beyond the peak do not have time to synthesize learning opportunities, or are otherwise stressed by the workload, such that their educational performance is poorer when compared to their colleagues with fewer encounters.

METHODS

We utilized an electronic medical record to count the number of patient encounters during post graduate year (PGY)-1 and PGY-2 of IM and pediatric residents starting with the cohort of residents that entered as PGY-1 in July 2006. We included data through the end of the academic year ending in June 2010. Exclusion criteria included failure to complete the academic year or the ITE during the subsequent year. For PGY-1 IM residents, 15 started in mid-year or did not complete the year and 23 did not take the ITE both years, with ten meeting both exclusion criteria, for a total of 28 exclusions out of 204 residents in the four cohorts entering the PGY-1 year (13.7 %). For PGY-2 IM residents, 11 residents of the 153 residents in the three cohorts that entered the PGY-2 year were excluded for not taking the ITE in PGY-2 or PGY-3 year (7.2 %). For pediatrics, three PGY-1 residents out of 43 (7.0 %) and six PGY-2 residents out of 32 (18.8 %) were excluded.

The ITE for IM is developed by the American College of Physicians (ACP) and cosponsored by the Association of Program Directors in Internal Medicine and the Association of Professors of Medicine.¹⁴ IM residents complete the ITE each October. The ITE for pediatric medicine is offered by American Board of Pediatrics.¹⁵ Pediatric residents complete the ITE in July.

The clinical rotations for each post-graduate year are found in Table 1. Each rotation lasts 4 or 5 weeks. In the Internal Medicine Program, PGY-1 residents had 1 month of elective experience, which could include inpatient consultation experiences (e.g., hematology or cardiology), outpatient consultation experiences in the subspecialty clinics (e.g. gastroenterology or pulmonology), research, or non-IM electives such as radiology or dermatology. Encounters in the Emergency Department (1 month) were not included as such encounters were not searchable in the electronic medical record. Experiences at hospitals not included in our electronic medical record system, such as international elective experiences, were not included. PGY-1 IM residents had 1 month of ambulatory care in addition to a once-weekly continuity clinic on one afternoon during non–ICU months. PGY-2 IM residents had 2 months of elective experiences.

Table 1.

Clinical Rotations (1 Month Each, Unless Specified)

PGY-1 IM	PGY-2 IM	PGY-1 Pediatric	PGY-2 Pediatric
Inpatient wards:	Inpatient wards:	Inpatient wards:	Inpatient wards:
• General medicine (3 months)	• Gastroenterology (1 or 2 months)	• General Pediatrics (3 months)	• Pediatric ICU (2 months)
• Medical intensive care unit	• Hematology (1 or 2 months)	• Neonatal Intensive Care Unit (3 months)	• Neonatal ICU
• Cardiology	• Pulmonology	• Hematology-Oncology	• Hematology-Oncology
• Cardiac intensive care unit	• Medical/Surgical ICU	• Newborn nursery	• Newborn nursery
• Neurology (3 weeks)	Consultation:	• Ambulatory Clinic	• Developmental Pediatrics
• Oncology	• Infectious Disease	Electives:	Electives:
Consultation:	• Nephrology	• 2 months	• 5 months
• Neurology (1 week)	• Endocrinology (2 week)
	Outpatient subspecialty:
	• Endocrinology (2 week)
Elective/Research:	• Allergy (1 week)
• 1 month	• Breast neoplasia (1 week)
Resident Continuity Clinic:	Resident Continuity Clinic:	Resident Continuity Clinic:	Resident Continuity Clinic:
• 1 month + ½ day per week	• ½ day per week	• ½ day per week	• ½ day per week
	Elective/Research:
	• 2 months
Not Counted:	Not Counted:	Not Counted:	Not Counted:
• Emergency Department	• Musculoskeletal (no notes written)	• Pediatric Emergency Department	• Pediatric Emergency Department

The Pediatric & Adolescent Medicine residency program trains both categorical pediatric residents and pediatric neurology residents each year. The categorical residents train for 3 years and sit for three ITEs, while the pediatric neurology residents sit for two ITEs during the pediatric-specific part of their training.

During the study period, PGY-1 residents in the Pediatrics Program had 2 months of elective experiences, which could include inpatient consultation experiences, outpatient consultation experiences, research, or non-pediatrics electives such as radiology or dermatology. They also were assigned to the Emergency Department (ED) (1 month), but these encounters were not included in the analysis, because the PGY-1 ED role is paired closely with a senior level resident and patient care encounters were not searchable from the electronic record for the PGY-1 resident. Each PGY-1 pediatric resident had a weekly continuity clinic experience in a routine ambulatory care clinic, and several months of various inpatient services. PGY-2 residents had 4–5 months of electives during the study period.

The ACGME duty hour requirements did not change during the duration of our study period, and neither IM nor pediatrics programs made significant changes to the resident call schedules during the 2006–2010 time frame. The internal medicine residency program placed a limit on the total number of patients on the medical services in late 2008, but the limit on the number of admissions by each individual intern and resident did not change during the study period.

Definitions

• Admission: Full history, physical, assessment and plan for a patient admitted to the hospital from the outpatient setting or emergency department. In the medical ICU and pediatric ICU, admissions also include patients transferred from the general medical unit to the ICU. Patients transferred from the ICU to the general medical unit were not considered admissions for the receiving resident. Transfer to a higher level of care was counted as an admission, because it prompts the care providers to reassess the differential diagnosis and perform new diagnostic evaluations. Transfers to a lower level of care generate a progress note rather than an admission note, and thus were not counted. We did not count daily progress notes in the inpatient setting for both logistical reasons (progress notes were not electronic for the duration of the study period) and because we wanted to examine the knowledge acquisition from the evaluation and assessment of undifferentiated patients, rather than the execution of care processes through the duration of a hospitalization. The programs did not utilize a night float system, so patients admitted by a resident were typically followed by the same resident for the duration of the hospitalization. Re-admissions to the hospital after discharge to the outpatient setting were considered new encounters.

• Inpatient Consult: Initial consultation by a resident on a consultative service (including elective experiences) regarding a patient admitted to the hospital. Subsequent progress notes were not counted as additional encounters.

• Outpatient Consult: History, physical, assessment and plan performed in the outpatient setting in a subspecialty clinic, such as Allergy or Endocrinology. Elective rotations were included. Subsequent visits were included.

• Resident Continuity Clinic: Encounters include patients seen for full medical examinations, as well as follow-up and acute visits in the primary care setting.

Data Collection

The patient encounters were counted in the electronic medical record by two authors (CPM, MBS) who were blinded from the ITE scores and conference attendance data. Attendance at core educational conferences was recorded by means of an electronic swipe card.

This study was deemed exempt by the Mayo Clinic Institutional Review Board, as it is related to education. IRB: 09–005144.

Statistical Analysis

Admissions were combined with inpatient and outpatient consultations for a combined hospital and subspecialty experience. Continuity clinic was analyzed independently. The total encounters from admissions, inpatient consultation, outpatient consultation, and resident continuity clinic were combined for a separate calculation of total encounters.

Four distinct multivariate linear regression models were used. For each of the two specialties considered, one model looked at the linear association between PGY-1 experiences (total of admit and consult notes, continuity clinic notes, and number of educational conferences attended) and percent correct on the PGY-2 ITE, adjusting for gender, medical school origin (accredited by the Liaison Committee on Medical Education (LCME) versus international medical graduate), and percent correct on the PGY-1 ITE. The second model within each specialty looked at the linear association between PGY-2 experiences and percent correct on the PGY-3 ITE, adjusting for gender, medical school origin, and percent correct on the PGY-2 ITE.

Within each of the four linear models, the possible interaction between admit/consult notes and continuity clinic notes was assessed. We assessed the Yerkes Dodson curve hypothesis by testing for improved model fit by including squared terms for both note types. A conservative alpha level of .01 was used to account for multiple comparisons. All calculations were performed using SAS statistical software (version 9.3; SAS Institute Inc., Cary, North Carolina).

RESULTS

Four PGY-1 cohorts and three PGY-2 cohorts had completed ITE testing by October 2010, of which a total of 176 IM residents (318 resident-years) and 40 pediatric residents (66 resident-years) met the inclusion criteria. The range of patient encounters is shown in Fig. 2 (internal medicine) and Fig. 3 (pediatrics), with the interquartile range demonstrated by the limits of the boxes. The average number of admission/consultation encounters for IM PGY-1 residents decreased from 185 in 2006–2007 to 155 in 2009–2010 (p < 0.001). The average PGY-1 ITE percent correct was 61.8 (SD 7.0) for IM and 60.7 (SD 7.1) for pediatric residents. The delta from PGY-1 to PGY-2 for IM was 5.7 (SD 4.8) and pediatrics was 9.9 (SD 5.3). The delta from PGY-2 to PGY-3 for IM was 4.8 (SD 4.4) and pediatrics was 3.0 (SD 5.9). The excluded residents did not differ from included residents in gender distribution, but the excluded PGY-1 residents were more likely to be graduates of international medical schools (IM p < 0.001, pediatrics p = 0.05). The average ITE-1 scores of included and excluded residents, when available, were not statistically different.

Figure 2.

Internal medicine volume of encounters by academic year. The number of admission/consult encounters by post-graduate year (PGY)-1 (gray) and PGY-2 (solid) internal medicine residents over academic years. Range in volume of patient encounters indicated by bars, with interquartile range (25th percentile to 75th percentiile) indicated by the boxes.

Figure 3.

Pediatric volume of encounters by academic year. The number of admission/consult encounters by post-graduate year (PGY)-1 (gray) and PGY-2 (solid) pediatric residents over academic years. Range in volume of patient encounters indicated by bars, with interquartile range (25%ile to 75%ile) indicated by the boxes.

The results of the multivariate analyses for internal medicine are found in Table 2. The total number of PGY-2 hospital and subspecialty encounters (admissions and consultations) was positively associated with PGY-3 ITE score (β = 0.02; p = 0.004), such that an additional 50 encounters was associated with an increase of 1 % correct on the subsequent ITE (Fig. 4). No association was found between total encounters in the first year of training and PGY-2 ITE score. Continuity clinic encounters were not statistically associated with change in PGY-3 ITE score. When PGY-2 inpatient, consultation and continuity clinic encounters were combined, no statistical association remained.

Table 2.

Multivariate Analysis for Internal Medicine Residents

Variable	PGY-2 ITE (n = 176)		PGY-3 ITE (n = 142)
	Estimate (SE)	P value	Estimate (SE)	P value
Male	0.643 (0.679)	0.34	1.028 (0.708)	0.15
IMG	0.136 (1.031)	0.90	−0.796 (0.934)	0.40
Prior year ITE % Correct	0.740 (0.049)	< 0.001	0.713 (0.056)	< 0.001
Prior Year Core Conferences	0.049 (0.018)	0.006	0.009 (0.015)	0.55
Prior Year Admit/Consult Notes	−0.013 (0.014)	0.36	0.023 (0.008)	0.004
Prior Year Clinic Notes	−0.033 (0.019)	0.08	−0.025 (0.017)	0.13

ITE in-training examination; IMG international medical graduate; SE standard error

Figure 4.

Graph of change in in-training examination (ITE) scores versus volume of admissions and consults for internal medicine (IM) post-graduate year (PGY)-2 residents. The plotted line is the correlation after adjustment for gender, medical school location, volume of encounters in the continuity clinic setting and conference attendance, thus it represents an “average” male graduate of a Liaison Committee on Medical Education (LCME)-accredited medical school with 68 % correct on ITE-2, 73 core conferences attended as PGY-2, and 145 encounters in continuity clinic.

Prior year ITE scores were significant predictors of subsequent year score (p < 0.001 for IM PGY-1 and PGY-2). For internal medicine, PGY-1 conference attendance was positively correlated with PGY-2 ITE score (β = 0.049; p = 0.006).

The results of the multivariate analyses for pediatrics are found in Table 3. No association between encounters and ITE score was found for pediatric residents for either of the years of training. Prior ITE scores were predictors of subsequent year score (p < 0.001 for PGY-2 ITE and p = 0.002 for PGY-3 ITE). Male gender was associated with a lower score in the PGY-3 year (β = −6.600, p = 0.02).

Table 3.

Multivariate Analysis for Pediatric Residents

Variable	PGY-2 ITE (n = 40)		PGY-3 ITE (n = 26)
	Estimate (SE)	P value	Estimate (SE)	P value
Male	0.200 (1.877)	0.92	−6.600 (2.582)	0.02
IMG	−5.800 (3.15)	0.08	2.257 (3.145)	0.48
Prior year ITE % Correct	0.756 (0.137)	< 0.001	0.746 (0.210)	0.002
Prior Year Core Conferences	0.006 (0.043)	0.89	−0.090 (0.050)	0.09
Prior Year Admit/Consult Notes	−0.005 (0.027)	0.85	−0.019 (0.0287)	0.52
Prior Year Clinic Notes	−0.059 (0.053)	0.27	0.077 (0.070)	0.29

ITE In-training examination; IMG international medical graduate; SE standard error

For both pediatrics and internal medicine, ITE score was not associated with international medical graduate status. No interactions between admit/consult and continuity clinic note counts were seen (all p > 0.20). There were no significant improvements in any of the four models fit by including squared terms for note counts, suggesting our sample is unsupportive of the Yerkes Dodson curve hypothesis (all p > 0.16).

DISCUSSION

Patient care and medical knowledge are ACGME core competencies for resident physicians. Currently, the core competencies are being further enhanced with milestones during the training process. The American Board of Internal Medicine draft framework to assess residents’ progression through the milestones includes the use of standardized tests such as the ITE.¹⁶ Currently, we do not know what volume of patient encounters is adequate to develop competency. Furthermore, the relationship between patient encounters and change in ITE score may not be linear (Fig. 1). It is conceivable that too few encounters will limit learning opportunities, while too many encounters results in inadequate time to read about and synthesize the educational pearls from the care of a patient. This inverted U curve was described by Yerkes and Dodson, and is associated with the learning of difficult tasks.¹³

In our study, there was a linear correlation in the IM PGY-2 year for the inpatient and subspecialty experiences, such that each additional 50 patient encounters was associated with an increase of 1 % correct on the ITE. The increase associated with the average of 263 patient encounters would be 5 % more correct on the ITE, which is consistent with the national average year-to-year increase in ITE score.¹⁷ The interquartile range for IM PGY-2 encounters is approximately 50. This suggests that based on clinical encounters, a resident in the 75th percentile of encounters would be expected to score 1 % higher on the ITE than a resident in the 25th percentile. Since change in ITE percentile is approximately 3.7 times percent items correct for scores ranging from 10 to 90 %, ⁵ moving across the entire range from the least encounters (127) to the most (346) would represent a maximum percentile increase of about 18 percentile points, which we believe is educationally significant.

IM PGY-1 residents had fewer encounters in the last 2 years of our study, likely related to the caps placed on the total number of patients cared for by each medical team. However, the range from the least encounters to most encounters by an IM PGY-1 resident was 131, which was smaller than the range of encounters for the IM PGY-2 residents, and the ranges for both cohorts of pediatric residents. Our null finding for the IM PGY-1 residents may be related to an insufficiently large range of encounters to detect a trend in ITE score.

The correlation between encounters and increased ITE score was only present in the IM PGY-2 year, which consists primarily of subspecialty rotations. This suggests that exposure to a wider variety of pathology may be associated with improved ITE score, consistent with similar results with medical student exposure to more core topics and subsequent standardized test scores.⁸ However, our current study did not track the specific diagnoses encountered by the residents, so further studies are warranted to investigate this hypothesis. Our results also reconfirm the positive correlation between core conference attendance and ITE score. In this cohort, the regression coefficient of 0.049 (P = 0.006) indicates that for every 20 conferences attended by an IM PGY-1 resident, we saw an increase of 1 % correct on the PGY-2 ITE, which is similar to the prior cohort of IM residents previously published.⁶

This study is limited to residents from a single academic institution that serves as a referral center. The findings may not apply to other academic centers or community training programs with different patient populations. In addition, resident sample size may not have been adequate to detect the correlation, particularly in the pediatric cohorts. The use of admission and consultation notes, but not subsequent progress notes, as a proxy for patient encounters may not take into account knowledge that is gained from discussions on team rounds or while caring for a patient admitted by another resident. The timing of the IM ITE in October results in residents being exposed to a portion of the subsequent year’s curriculum before taking the ITE, which may reduce the correlation of the ITE score with the experiences of a specific academic year. Finally, the range in volume of encounters by our residents may have been too narrow for a correlation to be demonstrated. Alternatively, there may also be a threshold effect, such that above a certain number of encounters, each additional encounter does not add significant knowledge. If such a threshold exists, the residents in our study may have exceeded that number of encounters, limiting our ability to detect an association.

In summary, we report the first demonstration of the correlation between resident patient encounters and subsequent medical knowledge. These findings may be important to residency programs, as they construct resident schedules to ensure that residents have sufficient exposure to pathology to meet competency milestones and are adherent to the duty hour requirements. In addition to conference attendance and structured reading, increasing the number of patient encounters may contribute to increased ITE scores. Future studies could investigate the effect of the duty hour requirements effective in 2011 on the volume of patient encounters, and also address how to balance interventions and patient care activities for residents with low ITE scores. Additionally, further studies could include training programs in different academic and community settings, as well as residency programs other than IM and pediatrics.