Assessing Physician Resident Contributions to Outpatient Clinical Workload

Background:

Graduate medical education is centered in hospitals despite a care system where patients mostly receive their care in an outpatient setting. Such gaps may exist because of inadequate funding for residency positions in community and hospital-based clinics.

Objective:

Determine if physician residents’ contribution to outpatient workload offsets their costs for supervision, salary, and fringe benefits as residents acquire skills to become independent practitioners.

Research Design:

VA’s electronic patient records from 2005 through 2018 were analyzed using generalized linear mixed models to estimate resident and staff contributions to workload in relative value units.

Measures:

Resident participation rate is resident contributed workload net of supervision as a percent of total clinic workload. Productivity is per diem resident workload as a percent of per diem staff workload. Efficiency is per dollar resident workload as a percent of per dollar staff workload. Progressive independence is annual rate of change in resident productivity.

Results:

Average participation rates varied by specialty from 6% to 22%, with 11% (primary care) and 13% (psychiatry). Productivity rates ranged from 21% to 94%, with 57% (primary care) and 61% (psychiatry). Efficiency rates varied from 0.63 to 3.81, with 1.69 (primary care), 1.89 (psychiatry). Progressive independence rates varied from 2.7%/year (psychiatry) to 39.7%/year (specialty care).

Conclusions:

Although residents rotating through most VA clinics generate revenue to cover their direct costs as they learn, some federal subsidies may be necessary to encourage hospital- and community-based clinics to accept residents from the less profitable primary care and mental health specialties.

Supervised patient care in teaching facilities is a critical component of graduate medical education (GME) where physician residents gain skills to become independent practitioners. However, GME is centered in hospitals despite a US health system that provides most of its care in outpatient settings.1 Some argue a GME outpatient gap exists because residents are financed through hospitals,2 and Medicare reimburses hospitals an additional $10b-$12b to cover their beneficiaries’ share of the hospital’s GME costs.3,4 In contrast, academic leaders generally accept outpatient care as important to the GME curriculum,5 offering residents more patients and a greater variety of cases,6 giving patients a higher quality of care,7,8 while permitting program directors to expand their capacity at a time when projected physician shortages for the U.S. are estimated to reach 139,000 by 2033.9

One significant barrier to GME expansion into outpatient care is costs for resident salaries and fringe benefits, plus staff time for education supervision activities.10–14 GME cost analyses often reveal staff productivity will decline when attending staff supervises residents in both ambulatory15–17 and hospital settings in medicine,18 family medicine,19 gastroenterology,20 radiology,21 and surgery.22,23

Offsetting these costs is the value of workload residents contribute to clinical workload. How much residents contribute is unsettled. An early study of outpatient care by Stern et al10 found resident productivity was too low to cover their direct costs, whereas a review of the literature found residents had little impact on billed charges, a slight decrease in the number of patients seen, and an increase in the amount of time the attending spent in clinic.24

Most of the optimism that resident productivity may recover GME direct costs comes from inpatient GME25,26 where the presence of residents were seen to change hospital costs only slightly,27 or reduced operating costs all together.28–31 These studies are often based on accounting data, from single facilities, limited time periods, and single specialty.4

In this paper, we measure residents’ contribution to patient care in hospital-based outpatient clinics and whether such workload is sufficient to cover their salary, fringe benefits, and clinical supervision costs. Rather than tracing cost accounts, we compare residents and staff inputs into the production of patient care in outpatient clinics at 125 VA medical centers offering GME for 9 specialty groups spanning 14 academic years. We estimate residents’ (1) participation in patient care, (2) productivity to produce workload net of supervision, (3) efficiency for their workload to cover their direct costs, and (4) progress to advance their skills towards independent practice.

METHODS

Setting

The study setting is the medical centers of the Department of Veterans Affairs. Accepting trainees since 1946, VA has become the largest “program of education and training of health personnel” in the United States.32 Administered by the Office of Academic Affiliations, VA offers affiliation agreements to sponsoring education institutions including 148 of 155 US LCME and 38 of 39 AOA-accredited medical schools. 33 These agreements cover the cost of salaries and fringe benefits for 11,600 residency positions filled by 47,500 rotating residents engaged annually in supervised clinical care at 125 VA medical centers at an estimated cost of $1.8 billion.3,33 VA’s statutory purpose is to enhance veterans’ access to VA care where residents are seen as both an immediate source of labor and a future pool to recruit new health professionals.34 Between 2005 and 2018, VA hosted 538,922 (33.3%) residents and fellows of the total 1,617,593 annual enrollment in US ACGME accredited programs.33,35 VA residents and fellows work together with 322,000 full-time professional and support staff as they care for 9 million veterans.36

Data

Outpatient encounter data were extracted from VA’s Corporate Data Warehouse from July 1, 2004 to June 30, 2018. More recent years were excluded to avoid confounds with the COVID-19 pandemic. Patient records identified primary and secondary providers by service date and clinic location. Common Procedure Terminology37 listed each procedure with their respective relative value unit (RVU) by year from the National Physician Fee Schedule Relative Value File. Provider specialty was classified by National Uniform Claim Committee (NUCC) Health Care Provider Taxonomy Code Set.38 Resident status for physician providers was indicated by searching text-based provider type, classification, and area of specialization fields for words such as “resident,” “fellow,” “student,” “trainee,” and “PGY”, or the person-class field classified the provider by a National Uniform Claim Committee (NUCC) education code.

Production units are clinics classified into 1 of 9 specialty groups (defined in Table 1): Primary Care, Surgery, Psychiatry, Neurology, Rehabilitation Medicine, Diagnostic Medicine, Emergency Medicine, Subspecialties of Medicine, and Specialty Care. Clinic-weeks are production units where residents, their supervisors, and other care providers classified by specialty are assigned to provide care for scheduled and walk-in patients during a given week. A patient encounter occurs when a patient has 1 or more visits to a clinic-week. A patient with multiple visits in the same clinic-week is counted as 1 encounter. The number of residents and providers engaged in patient care for a given clinic-week was quantified by provider-days computed as the sum of all days (or half days if a morning or afternoon shift) that providers spent in clinic that week. Estimates were based on encounter service date and time stamp recorded in the patient record.

TABLE 1.

Number of VA Clinic Facilities,¹ Clinic-Weeks,² Encounters, and RVU Workload, By Operating Status and Clinic Specialty⁸

Operating Status	P.C.	Surg.	Psych.	Neur.	Rehab.	Diag.	E.M.	S.C.	MSubS.
Clinic Facilities	130	128	130	126	130	130	123	130	130
with Residents3	120	107	113	93	109	124	107	117	106
	92.3%	83.6%	86.9%	73.8%	83.8%	95.4%	87.0%	90.0%	81.5%
who are Productive4	103	101	95	83	86	118	91	106	94
	85.8%	94.4%	84.1%	89.2%	78.9%	95.2%	85.0%	90.6%	88.7%
who are Efficient5	86	87	83	61	69	107	31	95	86
	71.7%	81.3%	73.5%	65.6%	63.3%	86.3%	29.0%	81.2%	81.1%
Clinic-Weeks (in 10,000s)	9.39	8.96	9.37	8.26	9.34	9.40	8.49	9.37	9.22
with Residents3	6.11	5.76	5.28	4.23	3.78	7.01	5.11	6.52	5.55
	65.1%	64.3%	56.3%	51.2%	40.5%	74.6%	60.2%	69.6%	60.2%
who are Productive4	4.90	4.70	4.48	3.89	3.03	5.96	3.87	5.65	4.63
	80.2%	81.6%	84.9%	92.1%	80.0%	85.0%	75.7%	86.6%	83.4%
who are Efficient5	3.88	3.69	3.51	2.58	2.43	5.14	1.40	4.98	3.70
	63.5%	64.0%	66.5%	60.9%	64.2%	73.3%	27.4%	76.3%	66.7%
Encounters (in 1,000,000s)	119.88	27.06	41.92	6.17	29.55	14.36	22.34	83.16	32.09
with Residents3	5.06	6.07	3.38	1.05	1.10	19.02	2.73	17.83	4.62
	4.2%	22.4%	8.1%	17.0%	3.7%	13.2%	12.2%	21.4%	14.4%
RVUs (in 1,000,000s)	272.20	184.58	121.87	16.91	56.28	394.95	59.68	263.62	179.73
Residents engaged6	13.62	62.32	10.99	3.30	4.00	56.51	7.15	64.76	26.19
	5.0%	33.8%	9.0%	19.5%	7.1%	14.3%	12.0%	24.6%	14.6%
Residents contributed7	25.10	20.42	13.50	2.60	2.75	95.47	2.79	48.46	21.56
	9.2%	11.1%	11.1%	15.4%	4.9%	24.2%	4.7%	18.4%	12.0%

¹ Clinic facilities are counted if they provide at least 1 patient encounter during the 14-year study period.

² Clinic-weeks include all unique patients treated and all procedures produced at a given outpatient clinic in a given facility during a given calendar week. There are a possible 94,640 clinic-weeks: 130 facilities × 14 years/facility × 52 weeks/year.

³ With Residents means a resident was assigned to a patient’s care as indicated by having been named as 1 of the care providers in the electronic health record.

⁴ Productive means the resident contributed a positive amount to clinical workload net of supervision as measured in RVUs.

⁵ Efficient means the resident's productive contribution net of supervision was sufficient to cover the resident's salary and fringe benefits.

⁶ Residents engaged includes RVUs of procedures listed in patient encounters where a resident was named as one of the care providers in the electronic health record.

⁷ Residents contributed is RVUs a resident is estimated to have contributed to workload net of supervision based on econometric methods explained in the text.

⁸Clinics are classified into 1 of 9 specialty groups defined as follows: While listing over 700 classifications, clinics were only included if physicians were engaged. Although not named here, listed clinics were often subdivided into Individual and Group sessions, Telehealth and In -Person, Hospital-based, Home-care, and Hospital-based-Home-Care, and by Patient Groups (Women, Homeless Veterans, Selected Disorders). Primary Care (PC) includes among others: General Medicine, General Internal Medicine, Family Practice, Post-Deployment Integrated Care, Health and Well Being Services, Hospital Based Home Care, Observation Medicine, Chronic Infectious Disease Primary Care, Women’s Primary Care Clinic, Primary Care Medicine, and Geriatric Primary Care, and Immunization. Surgery (Surg.) includes among others: General Surgery, Cardiac Surgery, Hand Surgery, Neurosurgery, Orthopedics, Plastic Surgery, Thoracic Surgery, Cardiothoracic Surgery, Bariatric Surgery, Surgical oncology, Vascular Surgery, Ambulatory Surgery, Observation Surgery, Obstetrics, Otolaryngology Surgery, Anesthesia pre-op consult, Pre-Surgery Evaluation – MD, Post-Surg Routine Aftercare, Spinal Surgery, and Ambulatory Surgery Services. Behavioral Health (Psych.) includes among others: Mental Health, Mental Hygiene, Drug and Alcohol Dependence, Neurobehavioral, Smoking Cessation, Observation Psychiatry, Mental Illness Outreach Psychiatry – MD, Post Traumatic Stress Disorder, Methadone Maintenance, Opioid Substitution, Opioid Safety Education, Sexual Trauma, Mental Health Primary Care, Psychosocial Rehabilitation, Gambling Addictions, and Mental Health Compensated Work and Supported Employment. Neurology (Neur.) includes Neurology and Observation Neurology. Rehabilitation Medicine (Rehab.) includes among others: Rehabilitation Medicine, Polytrauma, Polytrauma Transitional Rehabilitation Program, Physical Medicine and Rehabilitation services, and if a physician were present Recreation, Physical, and Occupational Therapy, Audiology, Spinal Cord Injury, Amputation Follow-up, Kinesiotherapy, Blind Rehabilitation, Traumatic Brain Injury, Outpatient Visual Impairment, Cardio-pulmonary rehabilitation, Wheelchair and Advanced Mobility, Observational Medicine Rehabilitation, Cardiac Rehabilitation, and Residential Rehabilitation Treatment Programs. Diagnostic Medicine (Diag.) includes among others: x-ray, E.E.G., E.K.G., Laboratory, Nuclear Medicine, Ultrasound, Echocardiogram, Sing Photon Emiss Tomography, Health Screening, Evoked Potential, Topographical Brain Mapping, Hypertension screening, Mammogram, Cervical Cancer Screening, PAP Test, Myocardial Perfusion studies, Positron Emission Tomography, Computerized Tomography Scans, Magnetic Resonance Imaging, Angiogram Catheterization, Magnetoencephalography, Computer Aided Design / Modeling Unit, Comprehensive Fundoscopy Exam, Diabetic Retinal Screening, Pathology, and Interventional Radiology. Emergency Medicine (E.M.) includes among others: Emergency Unit, Urgent Care, Emergency Department, Observation Emergency Room, and Crisis Emergency. Specialty Care (S.C.) includes among others: Oncology, Gynecology, Obstetrics/Gynecology, Ophthalmology, Proctology, Urology, Visually Impaired, Prosthetics/Orthotics, Chemo Clinic, Nutrition/Dietetics, Dermatology, Parkinson’s Disease, Genomic Care, Pediatrics, Multiple Sclerosis, Hospital at Home, War Related Illness and Injury Study Center, Electrophysiology Laboratory, Otolaryngology Clinic (non-surgery), Hemodialysis, Peritoneal Dial Treatment. Subspecialties of Medicine (MSubS.) includes among others: Cardiology, Gastroenterology, Hematology, Hypertension, Diabetes, Infectious Diseases, Nephrology, Rheumatology, Pulmonology, Cardiovascular Nuclear Medicine, Endocrinology/Metabolic, Liver, Hepatology Clinic, Renal/Nephrology, Chemotherapy Procedures Unit Medicine, Cardiac Catheterization, Cardiac Stress Test, Geriatric Clinic, Alzheimer and Dementia Clinic, Sleep Medicine, and Palliative Care.

RVU indicates relative value unit.

Workload is quantified by patients, encounters, procedures, or RVUs. Resident encounters are all clinic visits for which a resident is listed in the medical chart as 1 of the care providers. Resident engaged workload is the total RVUs corresponding to all procedures listed in resident encounters. Resident contributed workload is the increase (decrease) in the clinic’s total workload that would not have occurred but for the presence of the residents. Contributed workload is computed econometrically from patterns in how workload changed as the number of assigned residents changed over time.

VA residents’ salary and fringe benefits were computed as weighted averages by provider specialty from OAA’s financial records for academic year 2019–2020. First, we computed 296 salary plus fringe benefit rates for 8 PGY levels from among the 128 VA medical centers, 276 disbursement agents, and 318 academic affiliates. Fringe benefits included federal and local taxes, health, disability, and life insurance plans, retirement benefits, workers compensation, unemployment compensation, 21-day paid leave, and parking, lab coats, and other personal expenses. Next, we determined an average rate across the 8 PGY levels by specialty, aggregated specialty into 1 of the 9 outpatient specialty groups, and weighted by the 11,578.3 positions assigned to each 3584 facility-affiliate-specialty triplets across the 368 facility-affiliate pairs.

VA physician salaries were based on official pay tables effective February 16, 2020 that reflect a base, market, and performance pay for VA staff physicians grouped by specialty.39 Fringe benefits were computed at 30% of salaries.

Medical centers were also classified by VA on a 5-point complexity scale based on the mix of complex clinical programs, research dollars, number of residents, patient risk score, intensive care unit, and operative complexity level.40

Analytics

Estimates of resident contributions to workload were based on linear production models computed for each specialty S, facility clinic f, and academic year T:

$$W_{SfTt}={\alpha }_{SfT}+{\beta }_{SfT}\left[R_{SfTt}\right]+{\delta }_{SfT}\left[P_{SfTt}\right]+{\mathbf{\gamma }}_{SfT}^{\prime }\left[{\mathbf{X}}_{SfTt}\right]+v_{SfTt},$$ (1)

where W _SfTt is RVU workload produced, R _SfTt is total days worked by physician residents, P _SfTt is total days worked by staff physicians, and X _SfTt is a (k×1) vector representing the number of days worked by nurses, nurse practitioners, pharmacists, physician assistants, and psychologists, respectively, in specialty group S, at facility f, during academic year T for week t=1, 2, …, 52. α _SfT are parameter constants. β _Sft , δ _SfT , and (k×1) vector γ _SfT are coefficients representing mean per diem productivity rates, respectively, for residents, physician staff, and k-other provider types by service, facility, and academic year. v _SfTt is a zero-mean normally distributed random variate. Clustering productivity by specialty and academic year corresponds to VA’s signed annual affiliation agreements with its academic partners that permit residents to rotate through VA.

To account for changes in productivity as residents progress their clinical skills over the academic year, we set ${\beta }_{SfT}={\beta }_{SfT}^0+{\beta }_{SfT}^1\left[t^{⁎}\right]$ where $t^{⁎}=\frac{t-26.5}{51}$ ranges from $-0.5\mathrm{to}+\mathrm{0.5.}{\beta }_{SfT}^0$ is residents’ mid-year productivity, and ${\beta }_{SfT}^1$ is mean annual change in resident productivity. Accounting for reduced productivity whenever staff physicians supervise residents, we set: ${\delta }_{SfT}={\delta }_{SfT}^0-{\delta }_{SfT}^1\frac{\left[R_{SfTt}\right]}{\left[P_{SfTt}\right]}$ where ${\delta }_{SfT}^0$ is physician per diem productivity when no residents are present, and ${\delta }_{SfT}^1\frac{\left[R_{SfTt}\right]}{\left[P_{SfTt}\right]}$ is the reduction in physician productivity from supervisory education activities that increases with more supervised residents and decreases with more supervising physicians. Thus:

$$\begin{array}{l}{W}_{SfTt}={\alpha }_{SfT}+\left({\beta }_{SfT}^0-{\delta }_{SfT}^1\right)\left[R_{SfTt}\right]+{\delta }_{SfT}^0\left[P_{SfTt}\right]+{\mathbf{\gamma }}_{SfT}^{\prime }\left[{\mathbf{X}}_{SfTt}\right]\\ +{\beta }_{SfT}^1\left[t^{⁎}\right]\left[R_{SfTt}\right]+v_{SfTt},\end{array}$$ (2)

where $\left({\beta }_{SfT}^0-{\delta }_{SfT}^1\right)$ is resident productivity net of supervision at mid-academic year.

Resident contributions are computed by:

$${\mathrm{ℂ}}_{SfTt}=E\left[{\hat{W}}_{SfTt}|R_{SfTt}=r_{SfTt}\right]-E\left[{\hat{W}}_{SfTt}|R_{SfTt}=0\right],$$ (3a)

where $E\left[{\hat{W}}_{SfTt}|R_{SfTt}=r_{SfTt}\right]$ is predicted workload (Eq. 2) when resident days are set at the actual number of days worked, and $E\left[{\hat{W}}_{SfTt}|R_{SfTt}=0\right]$ is predicted workload when resident days are set at zero (no trainees). Residents are productive whenever $\mathrm{ℂ}\ge \mathrm{0}$ .

Resident productivity is workload a resident working an additional day would contribute net of supervision as a percent of the workload of an additional staff physician day would bring to specialty S, facility f, during academic year T:

$${\mathrm{ℛ}}_{SfT}=\frac{{\sum }_{t=1}^{t=52}\left(E\left[{\hat{W}}_{SfTt}|R_{SfTt}=r_{SfTt}+1\right]-E\left[{\hat{W}}_{SfTt}|R_{SfTt}=r_{SfTt}\right]\right)}{{\sum }_{t=1}^{t=52}\left(E\left[{\hat{W}}_{SfTt}|P_{SfTt}=p_{SfTt}+1\right]-E\left[{\hat{W}}_{SfTt}|P_{SfTt}=p_{SfTt}\right]\right).}$$ (3b)

Efficiency is the net workload an additional dollar spent on residents would generate relative to the workload that spending an additional dollar on physician staff would generate. Efficiency is computed by multiplying resident productivity ${\mathrm{ℛ}}_{\mathit{SfT}}$ , by the staff-to-resident-salary ratio $\left(\frac{C_{Sp}}{C_{Sr}}\right)$ , and indirect cost multiplier $\left(\frac{m_p}{m_r}\right)$ :

$${\mathrm{\epsilon }}_{\mathit{SfT}}{=\mathrm{ℛ}}_{\mathit{SfT}}\left(\frac{{\mathit{C}}_{\mathit{Sp}}}{{\mathit{C}}_{\mathit{Sr}}}\right)\left(\frac{{\mathit{m}}_{\mathit{p}}}{{\mathit{m}}_{\mathit{r}}}\right)$$ (3c)

where C _Sr and C _Sp are per diem salary and fringe benefits in dollars and m _r and m _p are indirect cost multipliers for residents and physician staff, respectively, in specialty group S. ε≥1 means residents’ contributions covered their costs and are “efficient”. If staff indirect costs are no smaller than resident’s $\left(m_r\le m_p\right)$ , then: ${\epsilon }_{SfT}={\mathrm{ℛ}}_{SfT}\left(\frac{C_{Sp}{m}_p}{C_{Sr}{m}_r}\right)\ge {\mathrm{ℛ}}_{SfT}\left(\frac{C_{Sp}}{C_{Sr}}\right)={\epsilon }_{SfT}^{⁎}$

Progressive independence is calculated as the change in productivity between the first and last week of a given academic year:

$$\begin{array}{l}{\mathrm{\wp }}_{SfT}=\left(\frac{E\left[{\hat{W}}_{SfTt}|R_{SfTt}=r_{SfTt}+1;t=52\right]-E\left[{\hat{W}}_{SfTt}|R_{SfTt}=r_{SfTt};t=52\right]}{E\left[{\hat{W}}_{SfTt}|P_{SfTt}=p_{SfTt}+1;t=52\right]-E\left[{\hat{W}}_{SfTt}|P_{SfTt}=p_{SfTt};t=52\right]}\right)-\\ \left(\frac{E\left[{\hat{W}}_{SfTt}|R_{SfTt}=r_{SfTt}+1;t=1\right]-E\left[{\hat{W}}_{SfTt}|R_{SfTt}=r_{SfTt};t=1\right]}{E\left[{\hat{W}}_{SfTt}|P_{SfTt}=p_{SfTt}+1;t=1\right]-E\left[{\hat{W}}_{SfTt}|P_{SfTt}=p_{fSTt};t=1\right]}\right)\end{array}$$ (3d)

where r _SfTt and p _SfTt are resident and VA physician staff days by specialty, facility, year, and week. ${\mathrm{\wp }}_{SfT}\ge 0$ means residents progressed during the academic year.

RESULTS

Overview

Across all 9 outpatient specialty groups, residents were engaged with 0.25b RVUs (16.1%) and contributed 0.23b RVUs (15.0%) of the total 1.55b RVUs produced between 2005 and 2018. Specifically, residents engaged 60.8m (12.0%) of 505.7m total patient encounters while rotating 493.6 k (60.3%) of 818.2 k clinic-weeks. Importantly, residents were productive (positive net contributed RVU) in 411.2 k (83.3%) and efficient (ie, net contributions sufficiently covered costs) in 313.0 k (63.4%) clinic-weeks.

Table 1 shows resident overall participation rates at the facility, week, encounter, and RVU level. Residents were engaged from 74% (Neurology) to 95% (Diagnostic Medicine) of operating facilities, and 41% (Rehabilitation) to 75% (Diagnostic Medicine) of all operating clinic-weeks. With the exception of Emergency Medicine, residents were productive (positive net contributions) in 80%–92% and efficient (net contributions covered costs) in 61%–76% of engaged clinic-weeks. Residents were engaged from 4% (Primary Care and Rehabilitation) to 21% (Specialty Care) and 22% (Surgery) of all encounters, and from 5.0% (Primary Care) to 33.8% (Surgery) of all RVUs. Residents’ contribution ranged from 5% (Emergency Medicine and Rehabilitation) to 24% (Diagnostic Medicine) of total RVU.

Resident-Engaged Clinic-Week

Table 2 shows how resident participation rates varied across the 9 VA specialty groups during weeks when residents were present and engaged in patient care. Neurology residents had the highest participation rates encountering 25% of patients and engaging in 31% and contributing 22% of RVU workload. In contrast, rehabilitation medicine had among the lowest participation rates, encountering 6% of patients, and engaging in 11% and contributing 7% of RVU workload.

TABLE 2.

Resident Participation, Productivity, Efficiency, and Progressive Independence Rates During Resident-Engaged Clinic-Weeks, by Clinic Specialty Group

Resident Metrics	P.C.	Surg.	Psych.	Neur.	Rehab.	Diag.	E.M.	S.C.	MSubS.
Resident participation*
Resident encounter rate (%)	5.7	25.1	10.1	24.6	6.4	12.5	14.8	22.0	16.0
95% CI	[5.4, 5.9]	[24.3, 26.0]	[9.7, 10.5]	[23.6. 25.5]	[6.0, 6.9]	[12.1, 13.0]	[14.0, 15.6]	[21.3, 22.6]	[15.4, 16.7]
10th percentile (%)	0.3	1.4	0.8	2.0	0.4	0.3	1.0	1.7	1.8
90th percentile (%)	10.0	40.3	17.3	37.2	12.4	21.7	30.6	34.6	31.5
Correlation† with
Engaged workload	0.98	0.94	0.95	0.96	0.78	0.91	0.98	0.98	0.96
Contributed workload	0.61	0.42	0.65	0.71	0.12	0.75	0.35	0.75	0.74
Engaged workload rate (%)	6.7	38.4	11.5	30.5	11.0	13.9	16.2	25.8	17.3
95% CI	[6.4, 7.1]	[37.1, 39.7]	[10.9. 12.0]	[29.2, 31.9]	[10.3, 11.6]	[13.3, 14.6]	[15.2, 17.1]	[24.9, 26.6]	[16.4, 18.1]
10th percentile (%)	0.3	1.1	0.5	2.3	0.5	0.2	1.0	1.6	1.2
90th percentile (%)	12.2	62.4	21.3	46.9	19.2	25.8	35.2	43.8	35.1
Correlation† with
Contributed workload	0.63	0.44	0.65	0.73	0.14	0.77	0.38	0.75	0.71
Contributed workload rate (%)	11.3	12.5	13.3	22.0	6.9	19.0	6.0	18.8	13.4
95% CI	[9.9, 12.7]	[10.3, 14.7]	[12.1, 14.6]	[20.5, 23.4]	[5.9, 8.0]	[17.6, 20.5]	[5.4, 6.6]	[17.5, 20.1]	[12.4, 14.4]
10th percentile (%)	−0.3	0.4	−2.1	−5.2	−4.0	−1.4	−2.4	0.7	−1.2
90th percentile (%)	21.9	22.4	21.5	40.4	11.7	32.6	16.3	34.3	23.3
Resident productivity (%)‡	57.0	54.2	61.3	51.0	60.4	76.3	21.1	93.9	60.2
95% CI	[52.7, 61.4]	[50.1, 58.2]	[57.3, 65.4]	[47.8, 54.2]	[55.6, 65.2]	[70.7, 81.9]	[18.9, 23.4]	[87.3, 1.01]	[55.3, 65.2]
10th percentile (%)	−16.0	9.7	−21.4	4.3	−40.9	−24.9	−1.9	16.9	−3.5
90th percentile (%)	120.1	98.3	115.5	74.3	108.4	127.6	40.6	167.6	102.5
Resident efficiency§	1.69	2.16	1.89	1.61	1.90	2.28	0.63	3.81	2.30
95% CI	[1.56, 1.82]	[2.00, 2.32]	[1.76, 2.01]	1.51, 1.71]	[1.75, 2.05]	[2.11, 2.44]	[0.56, 0.69]	[3.54, 4.08]	[2.11, 2.49]
10th percentile	−0.47	0.39	−0.66	0.14	−1.29	−0.74	−0.06	0.68	−0.13
90th percentile	3.56	3.93	3.56	2.34	3.42	3.80	1.21	6.80	3.91
Progressive Independence (%/y)‖	13.4	4.4	2.7	10.0	26.7.	12.7	3.3	39.7	20.3
95% CI	[13.3, 13.5]	[4.3, 4.5]	[2.6, 2.8]	[9.9, 10.1]	[26.6, 26.8]	[12.6, 12.8]	[3.2, 3.4]	[39.4, 39.9]	[20.1, 20.5]

^*Resident participation rates are computed in terms of patient encounters, RVU workload the resident engaged, and RVU workload the resident contributed to those clinics during those service weeks when the clinic was operating, seeing patients, and performing RVU procedures, and also residents were present and engaging in patient care activities. The Encounter rate is the percent of all patients treated in an operating clinic-week whose care had been engaged by a resident. Workload-engaged rate is the percent of workload (total RUVs of all procedures produced) that were engaged by a resident. Residents are said to have been engaged in patient care if they had been listed among the care providers in the patient’s electronic health record for that given encounter. Workload-contributed rate is the percent of total workload (RVU) that residents collectively contributed during an engaged clinic-week. Ratings >0 indicate residents collectively contributed net of supervision to weekly RVU workload. Tenth and 90th percentiles computed from facility rates averaged over 14-year study period.

^†Pearson correlations, P<0.001.

^‡Resident productivity rate is the average RVU workload net of supervision that would be contributed by adding 1 additional resident day divided by the workload that would be contributed by adding 1 additional physician staff day in the same clinic and facility.

^§Resident efficiency rate is RVU workload that would be contributed net of supervision for each additional dollar spent on the salary and fringe benefits of an average resident, divided by the RVU workload that would be contributed for each additional dollar spent on the salary and fringe benefits of an average physician staff member. Direct costs for resident salary and fringe benefits based on actual amounts VA paid to disbursement agents representing ACGME accredited programs for its residents and fellows. Fringe benefits ranged from 27.1% to 29.5% of the resident’s direct salary. VA physician staff salary based on midpoints from minimum and maximum allowed salary rates from 38 U.S.C. § 7431 and 84 FR 67340 (December 9, 2019). An efficiency rating of 1.00 or greater means that the average resident is contributing net of supervision an amount sufficient to cover their costs for salary and fringe benefits. The 10th and 90th percentiles are based on facility rates averaged over 14-year study period.

^‖Progressive independence is the annual change in the relative productivity rate during the 52-week academic year averaged over all facilities by clinic specialty group.

CI insicates confidence interval; Diag., Diagnostic Medicine; E.M., Emergency Medicine; MSubS., Subspecialties of Medicine; Neur., Neurology; P.C., Primary Care; Pych., Behavioral Health; Rehab., Rehabilitation Medicine; RVU, relative value unit; S.C., Specialty Care; Surg., Surgery.

Resident participation rates varied by measure. Resident encounter and engaged workload rates were essentially the same as both were indicators of a resident’s assigned case load. The correlation between engaged and contributed workload varied by specialty from 0.14 (Rehabilitation) to 0.77 (Diagnostic medicine) (Table 2), reflecting wide variability between contributed workload and total workload produced for encounters that had been assigned to residents. For Primary Care, Psychiatry, and Diagnostic Medicine, average contributed rates were greater than engaged rates indicating residents may have been responsible for more productive activities than what was indicated in the health record.

Resident participation rates also varied by facility. Averaged over the 14-year study period, residents in clinics operating at the 90th percentile were contributing 12% (rehabilitation) and 16% (Emergency Medicine) to 33% (Diagnostic Medicine), 34% (Specialty Care), and 40% (Neurology) of total RVU workload. In contrast, contribution rates in clinics operating at the 10th percentile ranged from −5% (Neurology) and −4% (Rehabilitation) to 0% (Surgery) and 1% (Specialty Care).

Facility complexity is 1 source for this variation. VHA facilities are classified from level 1 (highest) to 5 (lowest) complexity. Aggregated by facility and academic year, Table 3 shows resident RVU participation rates generally increased for facilities operating at a higher level of complexity. For Primary Care, facilities operating at the highest complexity level 1 had contributed rates of 14%, compared with only 3% for those at levels 3, 4, and 5. Diagnostic Medicine and Neurology had the widest gap in ratings, at rates of 25% and 23%, respectively, for level 1, and 2% and −7% for level 5, respectively. There were exceptions. For example, contributed rates in Rehabilitation Medicine tended to be the lowest for facilities at the middle level 3 complexity.

TABLE 3.

Annualized Facility-level Workload Contributed and Efficiency Rates, by Facility Complexity Levels^*

		Workload-Contributed Rate			Efficiency Rate
Complexity Level	n	Rate (%)	F†	P	Rate	F†	P
Primary Care
I. high -1a	515	14.3	42.4	<0.001	1.88	4.6	0.001
II. high -1b	279	10.2			1.46
III. high -1c	293	4.4			1.30
IV. medium -2	136	2.5			0.93
V. low -3	106	2.9			1.42
Total	1329	7.3			1.53
Surgery
I. high -1a	509	11.6	6.3	<0.001	1.63	1.9	0.11
II. high -1b	283	12.5			1.95
III. high -1c	320	8.4			2.52
IV. medium -2	119	5.0			2.48
V. low -3	34	8.0			2.11
Total	1265	9.6			2.00
Psychiatry
I. high -1a	506	11.8	7.3	<0.001	1.97	6.3	<0.001
II. high -1b	279	10.9			1.60
III. high -1c	264	5.1			1.18
IV. medium -2	123	6.3			1.30
V. low -3	66	7.5			2.22
Total	1238	9.3			1.67
Neurology
I. high -1a	500	24.2	14.8	<0.001	1.72	10.8	<0.001
II. high -1b	255	19.6			1.61
III. high -1c	191	11.2			1.13
IV. medium -2	62	−5.6			0.56
V. low -3	9	−2.7			0.12
Total	1017	18.5			1.50
Rehabilitation Medicine
I. high -1a	507	4.6	10.7	<0.001	1.81	4.48	0.001
II. high -1b	279	4.9			1.48
III. high -1c	249	3.3			1.28
IV. medium -2	81	5.3			1.48
V. low -3	50	6.1			1.06
Total	1166	4.5			1.59
Diagnostic Medicine
I. high -1a	481	24.6	28.3	<0.001	2.22	3.3	0.010
II. high -1b	263	25.4			2.62
III. high -1c	296	11.6			2.67
IV. medium -2	141	4.8			2.13
V. low -3	108	2.1			2.46
Total	1289	16.4			2.45
Emergency Medicine
I. high -1a	509	6.9	28.6	<0.001	0.70	4.6	0.001
II. high -1b	273	7.4			0.77
III. high -1c	294	2.7			0.52
IV. medium -2	123	−2.3			0.20
V. low -3	53	−9.1			0.13
Total	1252	4.5			0.60
Specialty Care
I. high -1a	527	19.3	20.5	<0.001	3.03	3.6	0.006
II. high -1b	289	19.5			2.81
III. high -1c	329	14.0			3.65
IV. medium -2	138	6.2			2.49
V. low -3	129	7.8			2.49
Total	1412	15.4			3.08
Medicine Subspecialties
I. high -1a	526	12.3	24.8	<0.001	2.02	2.6	0.037
II. high -1b	283	13.2			1.98
III. high -1c	294	4.7			1.41
IV. medium -2	126	4.5			1.65
V. low -3	43	11.9			2.59
Total	1272	8.9			1.88

^*The five-point complexity scale is computed by VA and is based on the mix of complex clinical programs, research dollars, number of residents, patient risk score, intensive care unit, and operative complexity level.

^† F statistic computed at n-5 and 4 degrees of freedom, where “n” is the total number of facility years reporting for the given outpatient clinic specialty group.

Resident Productivity and Efficiency

From Table 2, 1 additional resident is expected to contribute net of supervision between 21% (Emergency Medicine), 57% (Primary Care), and 61% (Psychiatry) to 76% (Diagnostic Medicine) and 94% (Specialty Care) of the workload that an additional staff physician contributes on the same service date and specialty group. As with participation rates, resident productivity varied widely among hospitals. For facilities whose mean rate ranked at the 90th percentile, productivity varied from 41% (Emergency Medicine) to 168% (Specialty Care). In contrast, residents in facilities whose mean rate ranked at the 10th percentile, productivity was negative in 6 of the 9 specialty groups, dropping to −41% (Rehabilitation). The wide variation in productivity rates underscores our finding that residents were productive in only 79%–95% of facilities and 76%–92% of resident-engaged clinic-weeks (Table 1).

Resident efficiency also varied by specialty group and facility. With the exception of Emergency Medicine, resident contributions to RVU workload were shown on average to be more than sufficient to cover their direct costs. An additional dollar invested in residents contributed 1.6 (Neurology) and 1.7 (Primary Care) to 2.3 (Medicine Subspecialty), and 3.8 (Specialty Care) times the RVU workload that an additional dollar invested in physician staff would make. Rates varied widely by facility. For facilities with mean rates operating at the 90th percentile, resident efficiency varied from 1.2 (Emergency Medicine), 2.3 (Neurology), 3.6 (Primary Care and Psychiatry) to 6.8 (Specialty Care). In contrast, facilities whose mean rate was at the 10th percentile were found to have efficiency rates <1.0 where resident contributions were not sufficient to cover their direct costs. These low rates underscore our finding (Table 1) that residents were efficient in only 29% (Emergency Medicine) to 86% (Diagnostic Medicine) of facilities, and 27% (Emergency Medicine) to 76% (Specialty Care) of resident-engaged clinic-weeks.

A source for this variation in efficiency rates is facility complexity, although the relationship is not always simple. Overall, clinics located in facilities operating at complexity levels 1 and 2 facilities tended to have the higher average efficiency rates, although there were exceptions. The association was not significant for Surgery, whereas resident efficiency in Diagnostic Medicine and Specialty Care clinics were highest when operating in level 3 facilities.

Progressive Independence

Table 2 shows an average gain in productivity during the academic year by specialty group. For Primary Care, resident productivity increased by 13.4% points during the academic year, with increases ranging from 3%/year (Emergency Medicine) and 4%/year (Surgery) to 20%/year (Medicine Subspecialties) and 40%/year (Specialty Care).

DISCUSSION

In this study of the largest system of teaching hospitals and clinics in the United States, we found residents rotating through outpatient clinics across different specialties made significant contributions toward billable services net of supervision. Our estimates were possible because we modeled resident production of clinical workload rather than traced resources through cost accounts. Overall, the value of resident contributions was sufficient to cover their salary, fringe benefits, and supervision costs in 63% of service weeks and 70% of clinics over a 14-year period. At the top tenth percentile, workload for Primary Care and psychiatry residents went as high as 3.6 times their direct costs, and 6.8 times for fellows in Specialty Care. However, such contributions seemed not to be at the expense of reduced learning as resident productivity across all specialties grew during the academic year consistent with progressive learning.41

These findings address concerns in the literature regarding the lack of financial feasibility in GME outpatient programs.10–14 While we focused on workload, GME program benefits to teaching residents in outpatient clinics includes, among others, expanding resident experience both in numbers of patients and variety of cases seen,6 improved care quality,8 possible reduction in burnout leading to improved staff recruitment and retention,34,42,43 exposure to team-based and patient-centered care,44 and staff professional achievement and satisfaction.45

These results also affirm the construct validity of these contributed workload estimates. Resident productivity estimates ranged between 21% and 94% of a VA staff physician as inexperienced trainees are expected to be less productive than their licensed counterparts. Average efficiency estimates ranged from 1.61 to 3.81 in all but 1 specialty (Emergency Medicine) as residents are expected to serve as lesser-cost sources of labor.25,26,46,47 The correlation between participation and progressive independence rates (r=0.30, P<0.001) is predictable of medical centers preferring residents from specialties who progress quickly to independent practice.41 Facility dependence on resident contributed workload was also associated with medical center complexity scores. Finally, positive associations were found between contributed and engaged workload, although it varied by specialty (0.12–0.75) as residents who were assigned to more frequent and difficult cases (engaged workload) are expected to also produce more work (contributed workload) depending on how resident case assignments were reported in the health record.

Resident participation rates and efficiencies were also correlated across specialties (r=0.60, P<0.001) suggesting VA medical centers sought the more profitable specialty and subspecialty resident than their less efficient counterparts. Residents in Specialty Care were 2.25 times more efficient than Primary Care or Psychiatry residents due to a 37% higher staff-to-resident-salary ratio and 65% higher resident productivity rating. Similarly, subspecialty residents were 1.36 times more efficient due to a 29% higher staff-to-resident-salary ratio and a modest 5% higher resident productivity rating. In both cases, the staff-to-resident-salary ratio played an important part to explain differences in efficiency. This has policy implications. If expanding GME to outpatient settings is intended to produce more primary care and mental health physicians, then a subsidy incentive may be required to make a less profitable specialties attractive.

These findings should be interpreted with caution. Not all procedures residents performed were captured by a CPT code. Interpreting differences across specialty groups is subject to aggregation bias. Trainee status was determined from text-entered provider specialty fields. However, total resident counts were within 98% of that reported in OAA’s annual Health Systems Survey where each medical center reports on the number of enrolled trainees.33 Another limitation is the absence of direct measures of care quality to assess whether higher productivity rates were achieved at the expense of patient outcomes. However, OAA strictly enforces policies governing supervision and monitors care outcomes. Another source is residents working more hours than their duty schedule allows and inflating productivity estimates. However, VA observes strict adherence to duty hour limits set by the Accreditation Council on GME, and strictly enforces supervision policies requiring attendings to be present when residents engaged patients.

To determine how estimates change when salary amounts came from other settings, we re-estimated efficiency rates using the 2020 Medscape survey of annual salaries of 17,461 physicians in 30 specialties48 and stipends of 1,659 US medical residents.49 Applying Medscape earnings data to VA estimated resident productivity rates, we found Medscape data lead to higher efficiency estimates for primary care (2.30 vs. 1.69), for a difference of +0.61. Similarly, we found Medscape data lead to higher efficiency estimates for Surgery (+1.38), Psychiatry (+0.78), Neurology (+0.60), Rehabilitation (+1.04), Diagnostic (+2.05), Emergency (+0.59), Specialty (+2.17), and Medicine Subspecialties (+0.65). Medscape higher rates reflect its higher staff-to-resident-salary ratio.

In conclusion, although residents in most VA outpatient clinics generated sufficient revenue to cover their direct costs as they learned skills toward becoming independent practitioners, some federal subsidies may be required to encourage hospital- and community-based clinics to accept residents from less profitable Primary Care and Psychiatry specialties. In addition, more research is needed to understand the underlying causes for the variation in resident productivity across clinics and specialties, and to understand how productivity may be associated with care quality, resident supervision, clinical learning, and resident program satisfaction, recruitment, and retention. Such information is critical if academic leaders are to extend GME training further into outpatient care settings.