Impact of inpatient caseload, emergency department duties, and online learning resource on General Medicine In-Training Examination scores in Japan

Abstract

Background

Both clinical workload and access to learning resource are important components of educational environment and may have effects on clinical knowledge of residents.

Methods

We conducted a survey with a clinical knowledge evaluation involving postgraduate year (PGY)-1 and -2 resident physicians at teaching hospitals offering 2-year postgraduate training programs required for residents in Japan, using the General Medicine In-Training Examination (GM-ITE). An individual-level analysis was conducted to examine the impact of the number of assigned patients and emergency department (ED) duty on the residents’ GM-ITE scores by fitting a multivariable generalized estimating equations. In hospital-level analysis, we evaluated the relationship between for the number of UpToDate reviews for each hospital and for the hospitals’ mean GM-ITE score.

Results

A total of 431 PGY-1 and 618 PGY-2 residents participated. Residents with four or five times per month of the ED duties exhibited the highest mean scores compared to those with greater or fewer ED duties. Those with largest number of inpatients in charge exhibited the highest mean scores compared to the residents with fewer inpatients in charge. Hospitals with the greater UpToDate topic viewing showed significantly greater mean score.

Conclusion

Appropriate ED workload and inpatient caseload, as well as use of evidence-based electronic resources, were associated with greater clinical knowledge of residents.

Introduction

Teaching hospitals are expected to provide high-quality educational environments to residents in order to enable them to attain a greater clinical knowledge and experiences. However, educational environments comprise many factors, including workloads, learning resources, clinical teachers, and hospital educational cultures. Among these factors, the programs of teaching hospitals can modify workloads and learning resources, whereas the behaviors of attending physicians and hospital educational cultures can be difficult to modify. Therefore, it is in our interest to evaluate how workloads and learning resources would affect the learning experiences of residents.

Our previous study showed that better, overall educational environment at teaching hospitals was associated with greater medical knowledge, but the study did not provide effects of each component of the educational environments on clinical performance of residents.¹ Among components of educational environments, inpatient workload may be a factor for better clinical performance, since clinical experience is considered to enhance learning in clinical medicine. Previous studies suggested that the cumulative number of inpatients in charge was associated with greater score of In-Training Examination (ITE) in internal medicine residents,² and the cumulative number of general pediatric admissions was correlated with an increase in ITE score in pediatric residents.³ However, these studies evaluated the cumulative number of inpatients as a workload for residents. A few studies have investigated the association between the average number of inpatients in charge at a time as a real-time caseload for residents and clinical knowledge in residents. This number may better capture the actual workload on daily basis. In addition, a few studies have investigated the association between the frequency of emergency department (ED) duties and clinical knowledge in residents. This number could reflect the actual exposure to emergency patients and could also be used as daily workload.

In this context, we aimed to investigate the association of clinical performance and objective workload using the average number of inpatients caseload in charge at a time and ED duties frequency.

Regarding learning resources, previous studies indicated higher scores of residents’ clinical performance for the amount of self-learning time in anesthesiologist residents ITE⁴ and in internal medicine residents^5,6 and self-learning by online resources⁷ in the Internal Medicine ITE (IM-ITE) and the amount of self-learning time. We also evaluated the online learning resources as a factor for greater clinical performance in residents since these resources are popular in modern clinical environments, in conjunction with the workloads.

Methods

Participants and data collection

We conducted a multicenter study involving postgraduate year (PGY)-1 and -2 resident physicians at 114 teaching hospitals in Japan. The participants were trainees of 2-year postgraduate rotation training programs, including internal medicine, emergency medicine, community medicine, and others (PGY-1 and -2), required for all residents regardless of their specialty, before getting into the specialty training programs (PGY-3 or later). The invitation to participate in the study was announced on the Japan Organization of Advancing Medical Education Program (JAMEP: Nonprofit Organization, Tokyo, Japan) Web site, monthly medical journal advertisement (Resident Note; Yodosha Co, Tokyo, Japan), and was sent to program directors of all 1,015 teaching hospitals in Japan certified by Ministry of Health, Labor, and Welfare, including 114 university hospitals and 901 community teaching hospitals.

The residency program directors of each hospital were required to assemble residents in a room at each hospital at a scheduled time and provide the General Medicine ITE (GM-ITE) for their residents. Each program director then collected the completed examination answer sheets and sent these back to us in an envelope we had provided. As the academic calendar in Japan starts on April 1 and ends on March 31 of the following year, the GM-ITE was conducted in March of the 2012 academic year. Immediately after the time of the GM-ITE, we provided a self-reporting questionnaire sheet about residents training loads (ED duties per month and average number of inpatients in charge at a time) and use of evidence-based electronic resources (DynaMed; EBSCO, MA, USA, and UpToDate; UpToDate Inc., MA, USA), obtaining informed consents from each participant. We received data about UpToDate usage from physicians in the hospitals who participated in our study. The UpToDate usage data were directly provided by UpToDate Inc and include the number of topic viewing. This study was approved by the institutional review board of the Mito Kyodo General Hospital, Mito City, Ibaraki, Japan.

Exposure variables

The primary exposure variables of our study were ED duties per month, average number of inpatients in charge at a time, and use of evidence-based electronic resources. Japanese PGY-1 and -2 residents are usually required to work in inpatient ward and in ED. Thus, we evaluated the association between the test score and workload in these two settings. The average count of ED duties per month was deemed as overall monthly ED workload. The average number of inpatients in charge at a time was used as overall daily inpatient caseload. The use of evidence-based electronic resources was indicated by regular online use of one of the following two most popular resources (that indeed account for vast majority of the electronic resources the participants used in their hospitals): DynaMed (EBSCO) or UpToDate (UpToDate Inc.) since these two resources were considered as the most popular online systems available in Japan. Other variables we collected and adjusted in the regression models include PGY.

Outcome variables

We developed and employed the GM-ITE using the similar methodology of developing the IM-ITE in the USA.⁸ The purpose of the IM-ITE is to provide residents and program directors with an objective, reliable, and valid assessment of each resident’s clinical knowledge in multiple-choice examination, and the mean scores of each program compared with that of its peers.^9,10 The aim of the GM-ITE is to assess clinical knowledge of PGY-1 and -2 residents. The test scores and the grade relative to the overall distributions of the test scores were provided for program directors of participating teaching hospitals and for each resident.

The GM-ITE examination included 100 questions testing a wide range of clinical knowledge, from clinical skills and practical medical knowledge to psychosocial care of patients, and was designed and written by a committee of experienced attending physicians organized by JAMEP. The examination gave a maximum score of 100 and a minimum score of 0; higher scores indicate a better performance of knowledge base of general medicine. Before conducting it, the examination question review was organized, and the content validity was confirmed by peer review of each member of the committee. A 3-hour fixed time limit was set for taking the examination. The GM-ITE was first implemented in 2011. Our study used data of the test that was performed in 2012, and an additional questionnaire was collected for the purpose of our study.

Statistical analyses

We investigated the association between the residents training loads, personal use of evidence-based electronic resources, and GM-ITE score by constructing a multivariable generalized estimating equations in order to account for the clustering of residents within each hospital. An exchangeable working correlation with identity link function and normal distribution was used for the analyses. Residents training loads in our study included ED duties per month and average number of inpatients in charge at a time. Evidence resource use was defined as personal use of online evidence summary (DynaMed and UpToDate). Continuous variables, such as the number of ED duties per month and average number of inpatients in charge at a time, were categorized into quintiles in order to allow potential nonlinear relationship with the outcome variable. The number of ED duties and inpatients in charge at a time were compared between residents who used evidence-based electronic resources and those who did not.

We also conducted a hospital-level analysis examining the association between the uses of evidence-based electronic resources and GM-ITE score aggregated at hospital level. We analyzed the association between mean hospital score of the GM-ITE and mean number of topic viewing per physician for UpToDate in each hospital for the preceding academic year with multivariable linear regression model. All statistical analysis was performed using IBM SPSS Statistics 21.0 (IBM Corporation, Armonk, NY, USA), and two-tailed P-value of <0.05 was considered as statistically significant.

Results

A total of 431 PGY-1 and 618 PGY-2 resident physicians in-training (290 women, 28%) underwent the GM-ITE and completed the survey (Table 1). As for the PGY-2 physicians, 159 (26%) were women and 459 (74%) were men. As for PGY-1 physicians, 131 (30%) were women and 300 (70%) were men. Among all participated residents, the mean score of the examination was 58.1 (standard deviation, 7.5). There were 114 teaching hospitals. Nine hospitals (7.9%) were university hospitals, while 105 hospitals (95.6%) were community hospitals. Evidence-based electronic resources were used by 790 residents (75.3%). There were 581 residents (55.4%) who had four to five times or more ED duties in an average month. There were 140 residents (13.3%) who cared 12 or more inpatients at any time in average ward rotation.

Table 1.

Characteristics of residents

Characteristics (N=1,049)	n	%
Sex
Men	759	72.4
Women	290	27.6
PGY
1	431	41.1
2	618	58.9
Hospital type
University	186	17.7
Community	857	81.7
ED duty per month (n)
0–1	72	6.9
2–3	166	15.8
4–5	581	55.4
6 or more	206	19.6
Number of inpatients in charge (at any time average)
0–5	246	23.5
6–8	395	37.7
9–11	224	21.4
12 or more	140	13.3
Use of evidence-based electronic resources
No	254	24.2
Yes	790	75.3

Note: Use of evidence-based electronic resources is defined as regular use of online evidence summary.

Abbreviations: PGY, postgraduate year; ED, emergency department.

After adjusted for PGY, multivariable generalized estimating equation analysis of total scores for all residents is shown in Table 2. Significant variables associated with the greater score included ED duties four to five times per month with mean score difference of 4.5 (95% CI, 2.7–6.3), number of inpatients in charge at 12 or greater times with mean score difference of 2.5 (95% confidence interval [CI], 0.96–4.0). Thus, residents with ED duties four to five times per month showed the greatest mean scores better than those with 0–3 or those with 6 or greater times of ED duties. Residents who routinely used evidence-based electronic resources showed 2 higher score (95% CI, 0.9–3.0) compared to those who did not use.

Table 2.

Generalized estimating equation analysis on total scores for all residents

Characteristic	Mean score difference			P-value
	Beta	95% CI
PGY-2 vs PGY-1	+1.62	0.72	2.53	<0.001
Use of evidence-based electronic resources vs no use	+1.99	0.93	3.04	<0.001
ED duty per month (n)
0–1	Reference
2–3	+3.97	1.96	5.98	<0.001
4–5	+4.51	2.73	6.28	<0.001
6 or greater	+2.90	0.95	4.86	0.004
Number of inpatients in charge
0–5	Reference
6–8	+0.74	−0.41	1.88	0.21
9–11	+0.92	−0.39	2.23	0.17
12 or greater	+2.50	0.96	4.04	0.001

Note: Use of evidence-based electronic resources is defined as regular use of online evidence summary.

Abbreviations: CI, confidence interval; PGY, postgraduate year; ED, emergency department.

The number of ED duties per month was not different between residents who used evidence-based electronic resources and those who did not use (4.5 vs 4.3, P=0.3). The mean number of inpatients in charge at a time was significantly greater among residents who used evidence-based electronic resources compared to those who did not use (8.8 vs 8.1, P=0.04). A multivariable linear regression model evaluating the association between mean scores and UpToDate review counts analyzed at hospital level revealed that UpToDate greater usage per physician was significantly associated with the greater score with mean score difference of 0.061 (95% CI, 0.007–0.114).

Discussion

We found that residents with ED duties four or five times per month showed significantly greater scores for the GM-ITE compared to those with 0 or once per month of the ED duties. Our results also showed that residents in charge of 12 or greater number (the group with the greatest number of caseloads) of inpatients at a time showed significantly greater GM-ITE scores compared to those with 0–5 inpatients in charge at a time.

A greater usage of evidence-based electronic resources at hospital level was significantly associated with the mean higher scores at a given hospital. Our results imply that greater workloads are needed for better performance of residents and that workload balance should be adjusted for achieving better performance as well as preventing residents’ burnout and adverse effects to patient safety.

Recently there has been a trend for limiting workloads for residents since there might have been association between greater workload and higher risk at burnout of residents. Previous studies on residents’ workloads showed that workload was related to burnout and medical error.^11,12 A study indicated that work hour limitation improved outcomes of patients in the teaching service.¹³ However, some studies, especially on surgical residents, suggested the opposite consequence, indicating that work hour limitation might lead to negative effects,¹⁴ null effects¹⁵ to patients’ outcome, and negative effects to the achievement of educational outcomes.¹⁶ Furthermore, programs with work hour limitation might not be attractive for residents with internal medicine.¹⁷

Many graduating students in Japan choose busy residency programs since they consider these are better educational environments for them. Our results also showed that a reversed U-shaped relationship between ED duties and performance of the GM-ITE scores. Thus, excessive workload in ED duties may have negative effects to residents’ performance. The appropriate workload balance should be aimed regarding designs for residency program for ED duties.

Residents who used evidence-based electronic resources exhibited greater scores of the GM-ITE in our study. Previous studies indicated higher scores of residents’ clinical performance for the amount of self-learning time.^4–7 Ecological studies on teaching hospitals also suggested that hospitals with evidence-based electronic resources had shorter mean days of their inpatients’ stay, lower mortality, and better quality of care.¹⁸ By the use of UpToDate immediately after inpatient rounds, 75% of clinical questions elicited by the rounds received satisfactory solutions and 37% of online reviewing led to change in clinical management.¹⁹

Although little time would be considered to be available for self-studying during busy residency programs, our results indicated that residents who used evidence-based electronic resources had improved clinical knowledge and were responsible for the significantly greater number of inpatients. Availability of evidence-based electronic resources at the point of care may be useful for supporting busy residents as a modern version of medical textbook and it could have better implication for the quality of patient care. Since our study provided association between the greater test score and appropriate workloads and use of evidence-based electronic resources, program directors and residents may be advised to consider setting appropriate workloads and using these resources.

Our study has several limitations. First, given that, only 1,049 of the 15,800 residents participated throughout Japan in this study, and they were not a random sample of the underlying population of all residents in Japan, our findings may not be generalizable to all residents in Japan. Program directors were responsible for the decision to participate in this examination and thus there might have been a bias for higher proportions for the participation among teaching hospitals with program directors with higher motivation and subsequently better programs. However, no residents who received the GM-ITE denied the participation for this survey and there was little space for bias toward the participation of better-performing residents. Second, with regard to the use of evidence-based electronic resources, better performers who used them efficiently might have had greater chance to provide care to the higher number of inpatients. Third, the emergency room duty per month and the number of inpatients in charge could be influenced by recall bias or cognitive bias since it is a questionnaire study. We use the term “on average” in the questionnaire about these items over the 1-year period before the test date. Fourth, we did not adjust potential confounders such as age and sex of individual residents in our model. However, these variables are usually not considered as significant variables associated with educational environments of teaching hospitals.

Conclusion

We found that appropriate level of workload, including ED duties and the inpatient number in charge, might have beneficial impact on the achievement of better performance of residents. In addition, the evidence-based electronic resources may be useful supporting tools for busy residents to get solutions for clinical questions raised by rounds. Our study may provide a benchmark for setting well-balanced workload for residents. Further study is needed for the interrelationships between workload, quality of life, and quality of patient care among residents.