Assessing the Training Costs and Work of Diagnostic Radiology Residents using Key Performance Indicators - An Observational Study

Abstract

Rationale and Objectives:

To quantify the costs and work of diagnostic radiology (DR) residents using the radiology key performance indicator turn-around time (TAT) as the outcome measure.

Materials and Methods:

In an IRB approved study, the annual cost of a DR resident was determined using salary, benefits and a cost allocation of faculty effort. The volume of cases reported in the 2015–16 academic year and median and interquartile range (IQR) turn-around time (TAT) for a trainee preliminary (C-P) or an attending final (C-F) radiology report were measured and stratified by time of day and patient location. Wilcoxon rank sum tests were used (significance, p-values < 0.05).

Results:

The annual cost of a DR resident was $99,109, 34% greater than direct salary/benefits and 27% of the direct salary/benefits cost of an attending. The total per minute cost of rendering care was $4.36 with both trainee ($0.70/minute) and faculty ($3.66/minute). Residents participated in 139,084/235,417 (59%) imaging studies. The C-P TAT was 74 (IQR, 27–180) minutes compared to 51 (IQR, 18–129) minutes C-F TAT of faculty working alone and C-F TAT of 213 (IQR, 71–469) minutes with a resident (p<0.001). The C-P TAT vs C-F TAT between 4pm-9am and weekends with residents is 44 (IQR, 18–119) minutes vs. 60 (IQR, 18–179) minutes without.

Conclusion:

The cost of training DR residents exceeds the salary and benefits allocated to their training. Residents increase the absolute professional labor cost of caring for a patient. Overall TAT is slower with residents but the care delivered by residents after-hours is faster.

Introduction:

A 2014 Institute of Medicine report, Graduate Medical Education that Meets the Nation’s Health Needs stated that the current funding structure for the training of physicians lacks accountability and measurable outcomes (1). The report recommends changes to graduate medical education (GME) financing and governance that could fundamentally restructure physician training. If academic medical centers (AMC) and teaching hospitals are to manage these impacts, quantification of the costs of residency training is essential.

Determining the true costs of health care is a challenge exacerbated in AMCs due to the intertwined missions of clinical care, education, and research (2). The overall effect of educational programs such as GME on the operating margin of AMCs has been debated for many years (3–6). Most AMCs are unable to accurately determine the true total cost of resident training or whether GME programs have a positive or negative effect on the institutional financial margin.

Recent studies have sought to define the benefits of the trainee workforce directly related to patient care. One study of pediatric residents in a Community Health Track setting found that a cost-benefit threshold was crossed when the team with trainees increased the volume of patients cared for in clinic from 9 to 12, concluding that it was cost-efficient to include residents in the workforce (7). A complimentary study of pediatric residents in a Longitudinal Outpatient Clinic setting demonstrated that the team with trainees yielded a greater average number of patient visits and revenue per faculty member but overall was associated with higher costs and lower operating margins than the faculty alone clinics (6).

The aim of this study is to better understand the costs and contributions of diagnostic radiology (DR) residents by determining the cost of the program and the residents’ contributions to care delivery via diagnostic imaging in the radiology department. This contribution is manifest though the direct patient care trainees perform in the process of evaluating and interpreting diagnostic imaging studies and rendering an imaging report. For this study, contribution is measured using the time from when an imaging examination is completed to when a report is available for medical decision making, the turn-around time (TAT), a common key performance indicator in radiology (8, 9). Resident contribution is compared to the cost and TAT of faculty working alone.

Methods:

This was an IRB approved observational study of GME trainees in a single DR training program. The DR residents consented to participation.

I. Determining the cost of the residency program

In 2015 the Department of Radiology at our institution consisted of 45 clinical faculty members, 24 residents, and 13 fellows. The department is in a public, academic university hospital in an urban setting, with Level 1 trauma and primary stroke center certifications, a National Comprehensive Cancer Network designation, and a full range of subspecialty and primary care networks.

The GME office developed a metric for calculating a detailed per-resident cost to understand and allocate the actual cost of GME that is distributed to the clinical departments (6). This included salaries, benefits and sources of funding. Departmental non-personnel expenses were captured with resident specific accounts that include travel, office supplies, printing, dues, professional development, space rental, business meals and entertainment, books and other resident related expenses. Expenses are covered by the primary department and the GME office. This also included allocated costs of faculty activity related to the supervision, education, assessment and recruitment of residents. For the purpose of this study, the costs are all attributed to the clinical department, and do not represent a shared burden with other entities, such as the hospital, the GME office, the state or third-party payors. In the radiology department, the faculty costs are those of the supervising physicians. There are no Radiology Extenders (RE) with educational or supervisory roles related to the delivery of diagnostic radiology services.

The total cost of imaging interpretation to the department is measured as

$$Totalcost=\sum _{n=s}^V\left(Cr\times I{r}_S+(Cr+Ca)\times R_S+Ca\times I{a}_S\right)$$

where V = Volume, s = an imaging study, Cr = Per minute cost of Resident, Ca = Per minute cost of Attending, Ia = Time for Attending to Interpret and Report, Ir = Time for Resident to Interpret and Report and R =Time for readout, note Cr = 0 and R = 0 when the Attending works alone.

II. Resident contribution to clinical care:

DR resident time was calculated from the duty hours reporting maintained in the institutional residency management system, (E*Value, Med-Hub, Minneapolis, MN). The Radiology Information System (Epic Radiant, Epic Systems Corporation, Verona, WI) was queried for the volume of cases the residents and faculty reported at the primary hospital setting and TAT, the time required to review a patient’s imaging study and generate a written report (9–11).

The workflow for a resident is initiated when an imaging study is classified as Complete (C) by the radiology technologist who acquired the imaging study (12). This workflow is described by Figure 1. The resident independently reviews the imaging and patient record and creates a draft (D) report that is not available outside the radiology workspace. After a readout, the resident incorporates faculty comments and publishes the Preliminary (P) report to the electronic health record. The P report is available for medical decision making by the patients’ care providers. The time to generate P is the C-P TAT. When an attending reviews and approves the report, the status changes to Final (F). This F report can be created by a faculty working alone or in conjunction with a resident. It is the definitive report in the electronic health record for medical care and initiation of the billing cycle. The time from when a patient’s images are available for review to when the report is finalized is the C-F TAT.

Figure 1:

Radiology exam interpretation workflow. Representations of typical radiology workflow for the process of reviewing, interpreting and generating a radiology imaging study report with a trainee (a), without a trainee (b) and afterhours, on nights and weekends (c).

Turn-around time variables C-P and C-F were collected as the primary outcomes. Additional variables collected for analysis included the patient location, divided into the Emergency Department (ED), Inpatient (IP) and Outpatient (OP). The time of day a report was created was also collected and, because of shift overlaps, stratified into distinct time intervals, Monday-Friday from 9am-4pm (routine) or Monday-Friday from 4pm-9am and Weekends (after-hours) for analysis. Studies with a C-F TAT less than 0 minutes or that exceeded 5 days (7200 minutes) were excluded, as these values are likely related to technical data transfer and verification issues or faculty availability issues such as vacation/meeting. Additionally, evaluation was limited to studies performed by the radiology clinical diagnostic subspecialty services that care for the highest volume of patients which are Abdomen, Cardiothoracic, Musculoskeletal and Neuroradiology. Interventional radiology was excluded. Diagnostic studies read by clinical fellows with a supervising faculty were excluded.

III. Statistical Analysis.

Hours allocated to clinical and educational activities during daytime service were summarized as N, mean and standard deviation (SD), and minimal and maximal values. Distribution of the TAT variables was checked against normality assumption using the Kolmogorov-Smirnov test. Time variables were summarized by resident presence, time of service, and patient location. Data were summarized as median and interquartile range (IQR) because of non-normal distributions. Wilcoxon rank sum tests were used to compare the median of the outcome variables between groups with and without residents. Statistical analyses were performed using Stata (Stata Corp. College Station, TX, USA) version 14. All tests were two-sided. P-values < 0.05 indicate statistically significant results.

Results:

I. Resident Base Cost and Daily Activity:

Table 1 shows the average annual cost per DR resident. An average annual DR faculty salary + benefits cost of $374,000 was derived from the actual clinical faculty salaries, excluding the Chair, and benchmarked to an average national salary in academics (13). The average hourly faculty cost is $219 (40 hours per week for 42.6 weeks/year, excluding 4.4 weeks of vacation and 5 weeks of continuing medical education). The direct faculty costs attributed to the training program included 0.3 FTE (12 hours/week) for the Program Director (PD) and Associate PD. Staff costs included 1.0 FTE (40 hours/week) for the Residency Program Coordinator. Annual direct faculty teaching costs were allocated based on delivery of didactic lectures, case conferences and the annual oral competency examinations that are part of the program’s structured assessment of resident medical knowledge. This teaching time cost is the sum of 600 hours (200 lectures at 3 hours per didactic lecture between preparation and delivery), 400 hours (200 case conferences at 2 hours per case conference between preparation and delivery) and 120 hours (10 faculty at 12 hours each) to the oral examination. Time allocation to resident recruitment was 384 hours (8 faculty at 48 hours each). Supervision of scholarly activity was approximated as 10 hours of faculty time/resident (14–16).

Table 1:

Annual cost of a Diagnostic Radiology Resident in US Dollars (n = 24 residents)

Expense Type	Annual Cost
Salary + Benefits	$ 74,059.00
Personnel	$ 21,989.43
Non-personnel	$ 3,060.50
Total cost per resident	$ 99,108.93

GME indicates graduate medical education; USD, United States dollars

Data are reported as the mean for all residents for the academic year July 1, 2015 to June 30, 2016. Personnel costs included dedicated program administration salaries and benefits (Program Director, Associate Program Director, Program Coordinator) and reimbursed and non-reimbursed faculty educational activity (committees, education, research mentoring, applicant reviews, and interviews). Non-personnel expenses included travel, office supplies, printing, dues, professional development, space rental, business meals and entertainment, books, and other resident-related expenses, accreditation fees, evaluation software fees, and central GME office administrative support.

The mean annual cost per DR resident was $99,108.93, 34% greater than the cost of salary and benefits alone.

According to E*Value duty hours logging, DR residents spent an average of 9.6 hours/day on service. The hourly cost of a resident is $42.14 based on a 48-hour work week during their 49 weeks of work per year (3 weeks of annual vacation).

II. Resident Involvement in Direct Patient Care

After data cleaning, 235,417 reports from the primary training institution that were finalized by faculty radiologists from 7/1/2015–6/31/2016 were available. Table 2 shows the distribution of reports and resident involvement in which the DR residents generated P reports for 59% of patient studies. The residents were involved in the care of 76% of ED patients, 59% of IP and 53% of OP. Figure 1, shows the workflow of report generation.

Table 2:

Percentage of studies reviewed and reports created by trainees based on the location of the patient

	Trainee	No Trainee	Total (%)
Emergency	36,595	11,300	47,895 (76%)
Inpatient	29,688	20,418	50,106 (59%)
Outpatient	72,801	64,615	137,416 (53%)
Total	139,084	96,333	235,417(59%)

The TAT for studies is significantly longer with residents (Table 3). The median C-F TAT with a resident is 213 (IQR, 71–469) minutes compared to 51 (IQR, 18–129) minutes without. This both statistically and clinically significant difference persists when comparing the resident C-P TAT of 74 (IQR, 27–180) minutes to the attending alone C-F TAT of 51 (IQR, 18–129). The difference in C-P and C-F TAT persists for all patient locations (Table 4). Although, the difference in median C-P vs C-F TAT for IP studies at 72 (IQR, 27–169) minutes compared to 73 (IQR, 20–164) minutes is not considered clinically important.

Table 3:

Time (in minutes) to produce preliminary and final reports with and without trainees

	Trainee Median (IQR)	No trainee Median (IQR)	P-value*
C-P	74 (27–180)	N/A	N/A
C-F	213 (71–469)	51 (18–129)	< 0.001
C-P vs C-F	74 (27–180)	51 (18–129)	< 0.001

^*Wilcoxon rank-sum tests were used

C-P = Complete to Prelim; C-F = Complete to Final

Table 4:

Time to produce preliminary and final reports based on the location of the patient

	Trainee	No trainee	p-value*
Emergency
C-P	26 (13–49)	N/A	N/A
C-F	177 (42–471)	13 (6–28)	< 0.001
C-P vs C-F	26 (13–49)	13 (6–28)	< 0.001
Inpatient
C-P	72 (27–169)	N/A	N/A
C-F	212 (70–430)	73 (20–164)	< 0.001
C-P vs C-F	72 (27–169)	73 (20–164)	< 0.001
Outpatient
C-P	146 (72–262)	N/A	N/A
C-F	224 (91–491)	59 (24–133)	< 0.001
C-P vs C-F	146 (72–262)	59 (24–133)	< 0.001

Wilcoxon rank-sum tests were used.

C-P = Complete to Prelim; C-F = Complete to Final

When the TAT is stratified by time of service delivery the C-P TAT with residents vs C-F TAT without residents during routine hours is longer at 122 (IQR, 55–224) minutes compared to 48 (IQR, 18–111) minutes (Table 5). After-hours the median C-F TAT with a resident is 247 (IQR, 59–540) minutes compared to 60 (IQR, 18–179) minutes without. Residents deliver faster care after hours, where the C-P TAT vs C-F without residents is 44 (IQR, 18–119) minutes vs 60 (IQR, 18–179) minutes. After-hours the residents operate with less direct faculty supervision. The TAT difference is likely due to the fact that the few faculty present are working both independently and reviewing resident work contemporaneously until 10 pm, while the resident is working alone from 10pm-7am and the dedicated readout may not be enforced.

Table 5:

Time to produce preliminary and final reports based on time of day

	Monday-Friday, 9 am to 4 pm			Monday-Friday, 4pm to 9am or Weekend
	Trainee	No trainee	P-value	Trainee	No trainee	P-value
C-P	122 (55–224)	N/A	N/A	44 (18–119)	N/A	N/A
C-F	192 (84–368)	48 (18–111)	< 0.001	247 (59–540)	60 (18–179)	< 0.001
C-P vs C-F	122 (55–224)	48 (18–111)	< 0.001	44 (18–119)	60 (18–179)	< 0.001

Wilcoxon rank-sum tests were used

C-P = Complete to Prelim; C-F = Complete to Final

III. Per minute cost of delivering patient care

Based on the average salary, the per minute professional personnel cost of a faculty is $3.66 and of a resident is $0.70. The per minute cost is not allocated over an entire C-F time span, because the actual time that either the resident and/or faculty interact with the patient’s study is not continuous. Because both a resident and faculty will participate in care, the total per minute cost with both is $4.36, an amount 16% greater than the cost without a resident. With these values, the total cost equation becomes

$$Totalcost=\sum _{n=s}^V\left(0.70\times I{r}_s+4.36\times R_s+3.66\times I{a}_s\right)$$

Discussion:

Overall radiology clinical care delivered with residents takes more time and costs more than care delivered by faculty alone. A potential offset of the cost is if faculty effort is streamlined by a resident’s participation in patient care. If the resident’s observations and conclusions are correct and comprehensive, then the faculty may be able to provide a more focused review with a resultant time savings. If the attending finalizes the resident’s report with minimal or no edits there is a potential savings. For example, if a faculty spends 8 instead of 10 minutes with an individual patient, because a resident has already invested 10 minutes, the total direct cost of imaging interpretation is $36.28, compared to the $36.60 cost if the faculty spent 10 minutes to care for that same patient.

An argument can be made that, as the resident progresses though the four years of training, competency and efficiency increases, thus providing a greater benefit to faculty than a resident in the initial years. However, members from all training years are distributed across the service lines, and thus, the least efficient and knowledgeable and most efficient and knowledgeable residents contribute to care and require supervision by a faculty in any given year. The question of this study is an assessment of the annual cost of a training program. Thus, a sub-analysis by year of training was not undertaken.

Shielding the faculty from effort that is not revenue generating, but that is critical to providing patient care in radiology, may offset some resident cost. Additional jobs that a resident may be performing that contribute to patient care on diagnostic services include protocoling studies (17, 18). These are not separately compensated tasks, nor readably measurable other than as a percentage of studies protocoled versus non-protocolled. Similarly, the time spent discussing imaging with care providers and technologists are not uniquely compensated tasks. However, because the ability to provide these services is predicated on the teaching and supervision provided by faculty, no separate allocation of benefit of these activities was performed.

A consideration in weighing the benefit of the additional cost of residents is if there is an improvement based on multiple patient interactions. Lauritzen et al (19) demonstrated that double reading chest CT examinations reveals interpretations discrepancies that would have an impact on patient management in up to 9% of cases. European guidelines from 2006 recommended double reading mammograms for optimal accuracy based on studies demonstrating a 5–15% improvement in sensitivity (20). A more recent mammography meta-analysis suggested that double reading increases operational costs, produces similar cancer detection rates, and does not decrease false-positive interpretations compared to single reading, arguing against a reliable benefit of this practice (21). These results cannot be directly extrapolated as the studies represent the effect when interpretation and re-interpretation is rendered by “equals” rather than the tiered relationship of residents and faculty. Additionally, since there was no collection of data like diagnostic accuracy or agreement, patient outcome measures, such as mortality or morbidity, or process measures, such as length of stay, the extrapolations of benefit that can be attributed to the DR residents’ contributions are limited.

It is difficult to separate the independent benefit of DR residents in relation to their cost when using the TAT metric. Differences in prioritization rules for clinical decision-making drive differences in TAT. Because of an institutional 45 min TAT goal for ED studies, these are often reviewed by the faculty as soon as a resident performs an initial review and then quickly changed to a P report. After-hours, the resident bypasses the D report step, and the P report is reviewed during the subsequent daytime shift by an attending, accounting for the shorter C-P TAT and longer C-F TAT. Outpatient studies are usually reviewed in batches during regular weekday hours. Thus, the C-P TAT is longer for the patient whose study was drafted by the resident at 9 am than that reviewed at 11 am, if the review with the attending occurs at 11:15 am, and reports are changed from D to P at 11:45. This time lag will exist even if the resident and faculty spent the same amount of time with each patient.

An advantage to the workflow involving residents occurs when assessing the impact in an 24 hours a day, 7 days a week (24/7) environment (22). Afterhours C-P TAT is faster than the independent faculty C-F TAT. Shifting to a 24/7 model of in-house attending radiology coverage has been shown to reduce C-F time from a mean of 9.1 hours to 1.7 hours, while reducing the volume of resident cases, diminishing the resident’s perception of autonomy and reducing satisfaction with the educational experience, even when acknowledging that the teaching received from in-house faculty is superior (23). Because labor costs of residents are so much less than faculty labor costs, the maintenance of resident independence represents a potential cost savings that could be factored into negotiations with hospitals in relation to 24/7 coverage.

An alternative workforce to residents are Radiology Extenders (RE), including Nurse Practitioners, Physician Assistants and Radiology Assistants (24). Multiple recent studies have demonstrated benefits of these providers in radiology practices (25–27). Recent and proposed changes in Medicare rules will likely increase the role and opportunity for RE in both diagnostic and interventional practices (24). The costs of delivering care to patients with these professionals might be lower to the system, however their participation does not alleviate the need to train and develop the future physician workforce. Similarly, there is a cost associated with the training and supervision of those providers that would also be borne by health care systems. An AMC, with the training mission, is unlikely to exchange residents for RE. However, the methods used in this study could be used prospectively to estimate the cost and potential revenue gain of service delivery expansion using RE.

TAT and clinical productivity are well-accepted radiology performance indicators, on par with other common metrics such as financial strength, patient satisfaction, regulatory compliance, and academic performance in the AMCs (8, 28). However, these metrics explain little about the impact a given radiologist has on the quality of care delivered or patient outcomes. The assumption is that by making information available in a timely manner the patient will benefit. These should be more appropriately recognized as operational metrics than an assessment of performance of an individual provider and are limited in the task of measuring the benefit of residents in patient care. However other performance metrics, like hand hygene, or outcome metrics like contrast reactions, or patient falls (8) are similarly inadequate to the task of measuring the work of radiologists, whether faculty in an AMC, an independent provider in a private practice or a resident.

Resident independence, while an area where a program potentially minimizes cost, by using less expensive resources, does expose potential risk because of the longer time gap between when the resident issues an opinion that is available to drive clinical decision making and the final assessment. This initial opinion by the resident may be in error which could contribute downstream costs to the health care system. While of concern, multiple studies have demonstrated that the major error rate by radiology residents is no higher than the error rate documented in peer review of board-certified practicing physicians (22, 29, 30). Indeed, one study has shown that fewer addenda are made to reports when the initial study is interpreted by a resident than a staff physician (22, 29). Additionally, the argument has been made that the opportunity to provide care with graded independent responsibility is critical to the development of the skills and knowledge that is necessary to eventually practice outside the scope of a training program (31).

It is important to note that there are differences in the DR resident experience from other previously studied training specialties that may contribute to costs. DR residents average a 48-hour work-week, well below the recommended cap of 80 hours, and below the time observed in other training programs (6, 32). This translates to a higher per minute cost of care delivered by a DR resident. The volume of patients that a radiologist and by extension a DR resident interacts with is significantly greater than that of other medical specialties (33). The duration of time that a radiologist spends with a single patient is also shorter. Our residents spend a significant amount of time receiving education that is independent of direct patient care via lectures and case conferences. These differences make it difficult to directly extrapolate the observations from this study to other GME training programs.

In the model employed in this study, the faculty cost associated with education directly increases the cost of care delivered by a resident. This teaching investment competes with the time that the faculty is available to deliver patient care. Reducing protected time for education would have the effect of both reducing the direct cost of the resident to patient care and increase the hours faculty are available to care for patients. Eliminating the training period altogether for a model where physicians begin to practice independently at the completion of medical school would have unknown impacts. The consequences of such a shift in practice could come at the expense the quality care of future patients if physicians are less prepared for the gamut of pathology for which they will diagnose and treat in practice (32, 34, 35).

There are several limitations of this study. The study was designed to be very narrow in scope, only addressing costs allocations related to salary and benefits and report TAT. The data is from a single institution, and thus is only directly applicable to the experience and costs in this institution. Including the data from multiple training programs could increase generalizability. Not all costs associated with the work of administration of a residency training program were captured, such as costs of GME office staff, Program Evaluation Committee and Clinical Competency Committee meetings, among others. Inclusion of such costs would further increase the per minute cost of the resident. The time to care for an individual patient was not directly observed in this study. No independent allocation of costs or assessment of benefit was made to many of the other missions in which residents participate, such as medical student teaching, institutional and professional society service or research and discovery. The analysis was not comprehensive and did not include costs that were not as easily quantifiable such as variation between residents by year of training and between faculty, and the contributions of residents in non-interpretive tasks, that might be associated with cost savings or increases.

Conclusions:

In summary, the cost of training DR residents exceeds their salary and benefits. With residents, the delivery of patient care is slower and their involvement increases the professional labor cost. The timely care provided by DR residents after-hours settings may offset some of their overall costs. Understanding and accounting for these costs and the conditions of the training environment must be part of financial and strategic operating plans for AMCs.

Abbreviations:

AMC

Academic medical centers

GME

Graduate medical education

DR

Diagnostic radiology

C-P TAT

Complete-Preliminary turn-around-time

C-F TAT

Complete-Final turn-around-time

ED

Emergency Department

IP

Inpatient

OP

Outpatient

D

Draft report

P

Preliminary report

F

Final report

IQR

Interquartile range

FTE

Full time equivalent

RE

Radiology Extender