Differences in Preventive Health Quality by Residency Year

Abstract

Background

It is assumed that the performance of more senior residents is superior to that of interns, but this has not been assessed objectively.

Objective

To determine whether adherence to national guidelines for outpatient preventive health services differs by year of residency training.

Design

Cross-sectional study.

Participants

One hundred twenty Internal Medicine residents, postgraduate year (PGY)- 1 and PGY -2, attending a University Internal Medicine teaching clinic between June 2000 and May 2003.

Measurements

We studied 6 preventive health care services offered or received by patients by abstracting data from 1,017 patient records. We examined the differences in performance between PGY-1 and PGY-2 residents.

Results

Postgraduaute year-2 residents did not statistically outperform PGY-1 residents on any measure. The overall proportion of patients receiving appropriate preventive health services for pneumococcal vaccination, advising tobacco cessation, breast and colon cancer screening, and lipid screening was similar across levels of training. PGY-1s outperformed PGY-2s for tobacco use screening (58%, 51%, P=.03). These results were consistent after accounting for clustering of patients within provider and adjusting for patient age, gender, race and insurance, resident gender, and number of visits during the measurement year.

Conclusions

Overall, patients cared for by PGY-2 residents did not receive more outpatient preventive health services than those cared for by PGY-1 residents. Efforts should be made to ensure quality patient care in the outpatient setting for all levels of training.

Resident physicians are expected to gradually increase competency, building complex frameworks of knowledge and skills from clinical experience under the guidance of attending physicians.¹ A challenge for residency programs, however, is assessing and measuring trainee performance.² Although several evaluation tools exist to assess a medical resident's knowledge, skills, and attitudes,³ evaluation of performance is often subjective, may not reflect the true level of competence, and frequently uses an implicit expectation of expertise based on the residents' level of training. It is assumed that the performance of more senior residents is superior to that of interns who have not advanced through the training process. However, their performance is not routinely assessed based on objective measures that are applicable across different levels of experience.

To enhance the quality of health care for the American public, the American Board of Internal Medicine (ABIM) establishes high standards for certification of physicians in training. The ABIM requires Internal Medicine residency programs to evaluate their residents' performances in a variety of ways, including reviewing clinical documentation for format, quality, accuracy of assessment, and appropriateness in both the inpatient and outpatient settings.⁴ Additionally, the Accreditation Council for Graduate Medical Education (ACGME) recently developed and implemented 6 areas of competency that residents must demonstrate during their training. One of these competencies, Practice-Based Learning and Improvement, specifically addresses efforts to improve patient care practices and to analyze and evaluate residents' clinical experiences.⁵

Within the context of an ongoing quality improvement project at our institution, designed to fulfill the requirements of the ABIM and ACGME, we developed a chart abstraction tool to assess our postgraduate year (PGY)-1 and PGY-2 resident performances in their outpatient continuity clinic experience. Prior to the implementation of the quality improvement project, we measured the differences between PGY-1 and PGY-2 resident performances in terms of evidence-based quality of care guidelines using standardized chart abstraction of over 1,500 patient charts. Our aim was to examine potential differences in the performance of preventive health services by year of residency training.

METHODS

Study Design

The Preventive Health Achievable Benchmarks Curriculum was implemented at the University of Alabama at Birmingham (UAB) in the fall of 2003. The curriculum uses the Achievable Benchmarks of Care (ABC) audit and feedback methodology⁶ to provide feedback to residents on their performance of preventive health services in their ambulatory continuity clinic. The goal of the curriculum is to improve the quality of preventive health care services provided to patients and to expose the residents to the concept of practice-based quality improvement. For the analysis presented in this manuscript, cross-sectional data on the baseline performance of preventive health care screening and counseling were abstracted from the charts of patients seen by PGY-1 and PGY-2 residents in their ambulatory continuity clinic prior to the implementation of feedback. The UAB Institutional Review Board approved collection of anonymous patient data, linked and aggregated at the resident level.

Setting and Patient Population

The UAB Russell Ambulatory Clinic is located in the University Hospital in downtown Birmingham, Alabama. The clinic serves a low-income, low-education adult patient population, with a mean age of 43 years, 62% female, and 61% African American. PGY-1 and PGY-2 residents attend this continuity of care clinic, on average, 1 half-day per week. Clinic notes are dictated and stored as paper charts in the clinic.

Resident Sample

Two sets of cross-sectional data were collected. PGY-1 and PGY-2 residents participating in the curriculum in the June 2002 to May 2003 academic year were included. To further assess the consistency of the results over time, we abstracted patient charts from PGY-1 and PGY-2 residents for the June 2000 to May 2001 academic year.

Defining Preventive Health Performance Measures

We identified a set of 6 performance measures. We began with all preventive health measures as defined by the Health Plan Employer Data and Information Set (HEDIS^®)⁷ or included in the United States Preventive Services Task Force Guidelines.⁸ We then ranked indicators on a set of criteria including relevance to residency education, ability to abstract data accurately, perceived variability on the measure, and ability of the residents to change behavior based on feedback of the measure. We also focused on indicators that were driven by individual resident performance and not influenced by the clinic system or nursing factors. After ranking all indicators considered, we chose to focus on 6 indicators (Table 1), and measured them in the appropriate patients following the national guidelines. In agreement with guidelines, we obtained information for colon cancer screening by both fecal occult blood testing (FOBT) and invasive colon cancer screening.

Table 1.

Definitions of Preventive Health Performance Measures

	Definition	Ideal Candidate Population
1.Colon cancer screening
a. Fecal occult blood testing (FOBT)	Proportion of patients with FOBT obtained, scheduled, or offered in the past year	All patients aged over 50 years
b. Invasive colon screening	Proportion of patients with (1) flexible sigmoidoscopy or double-contrast barium enema within the past 5 years OR (2) colonoscopy within the past 10 years obtained, scheduled, or offered	All patients aged over 50 years
2. Breast cancer screening	Proportion of women with mammogram obtained, scheduled, or offered during the prior 2 years	Female patients aged over 50 years
3. Lipid screening	Proportion having been screened for high cholesterol within the past 5 years	All patients with (1) diabetes, (2) multiple risk factors for cardiovascular disease, (3) or age over 35 years for men, and 45 years for women
4. Pneumococcal vaccination	Proportion with chart documentation of ever having received or being offered at least 1 Pneumovax vaccination	Patients over 65 years or those under 65 years with chronic diseases (alcoholism, cardiac, renal, liver, or pulmonary disease, diabetes, HIV, lymphoma, organ transplant, or on steroids) or asplenism
5. Tobacco use screening	Proportion of patients screened for smoking within the past year	All patients
6. Advising tobacco cessation	Proportion of smokers who are advised to quit smoking within the past year	All patients who are screened and identified as current smokers

Note that for 3 of the measures (colon cancer screening, breast cancer screening, and pneumococcal vaccination), the indicator recognized documentation in the chart of an offer of the preventive service as adherence to the guideline, regardless of actual delivery of the service. Documentation of offer was also accepted if there was documentation of the service having been received. For the other 3 measures (lipid screening, tobacco use screening, and advising tobacco cessation), documentation of the preventive service being received was required.

Medical Record Abstraction

Using the electronic scheduling system in the UAB Russell Ambulatory Clinic, a patient schedule list was generated for all patients seen within the calendar year for each resident involved in the study. Charts of all patients seen at least twice by a single resident during the academic years June 2002 to May 2003 and July 2000 to June 2001 were abstracted. Research assistants used the customized MedQuest tool^9,10 on a laptop computer in the clinic to abstract data from the charts based on a standardized protocol. All components of the patient charts were reviewed during abstraction. Patient demographics including age, sex, insurance, and number of visits during the calendar year were collected. Definitions for each performance measure were embedded behind the indicator for reference. The research assistants were trained for 2 weeks using a random set of charts not included in the pool for final data collection. Research assistants were supervised and charts were double abstracted during training to assess for agreement and to identify systematic errors. Double abstraction was also used for a 5% sample to monitor quality during follow-up on a monthly basis. Any discrepancies between the primary abstractor and the second abstractor were adjudicated by group review. After adjudication, the mean error rate for the primary abstractor was less than 2%.

Resident Data

Resident-level data including medical school of origin, post-graduate year (PGY-1 vs PGY-2), categorical versus primary care track, intended future practice (primary care vs subspecialty), gender, and age were available in house staff office records. These data were linked to the chart abstraction data, and unique resident identifiers were then removed.

Analyses

Separate analyses were conducted at the patient level for each quality indicator. For each indicator, a population of ideal candidate patients, for whom the preventive measure would almost always be indicated, was identified (Table 1).¹¹ For these ideal candidate populations, we calculated the proportion of patients who had documentation in their chart of the preventive health service being offered or received. We compared the proportion of patients who were offered or who received preventive services among patients seen by PGY-1s with patients seen by PGY-2 residents. In the analysis, we accounted for the nesting of patients within providers and adjusted for multiple confounders using generalized estimating equation (GEE) logistic regression models. An exchangeable working correlation matrix was assumed for the GEE models (implemented with STATA SE 8.0 statistical program). For those indicators for which differences were significant in the univariable analyses, additional multivariable adjustments for resident and patient demographics were conducted.

RESULTS

Characteristics of Residents

A total of 120 Internal Medicine residents, including 11 medicine-pediatric residents, were assessed. Fifty-three (44%) were PGY-1s, and 67 were PGY-2 residents. Thirty-two percent (N=37) were in the primary care track; 31% were female. Comparing the trainees from 2000–2001 academic year (N=54) to the 2002–2003 academic year (N=66), a similar proportion planned to enter primary care (31% vs 32%). The 2002–2003 trainees were more frequently female (35% vs 25%) and included more PGY-1 residents (30% vs 22%), but the differences were not statistically significant.

Comparing Performance of PGY-1 Versus PGY-2 Residents

One thousand seventeen charts were abstracted, with a mean of 8.5 charts abstracted per resident. PGY-2 residents did not significantly outperform PGY-1 residents on any of the indicators measured (Table 2). The difference for tobacco use screening was significantly different when comparing PGY-1s versus PGY-2s (58% vs 51%, unadjusted odds ratio 1.36 (95% confidence interval (CI) 1.003, 1.84)). After adjustment for clustering within provider, for patient age, gender, race and type of insurance (Medicaid or no Medicaid), resident gender, primary care versus categorical track training, and number of visits during the measurement year, patients seen by PGY-1 residents were again more likely to be screened for smoking (odds ratio (OR) 1.48 (1.06 to 2.07), P=.023) as compared with those seen by PGY-2 residents.

Table 2.

Receipt of Preventive Services, Patients Seen by PGY-1 (N=53) Versus PGY-2 (N=67) Residents: An Adult University Internal Medicine Continuity Teaching Clinic, 2000 to 2001 and 2002 to 2003

	No. of Ideal Candidates	Proportion of ideal candidate patients receiving preventive measure
		Overall (%)	PGY-1 Residents' Patients (%)	PGY-2 Residents' Patients (%)	Unadjusted Odds Ratio (95% CI) P value*
Pneumococcal vaccination	692	61	63	59	1.14 (0.79, 1.64) [.47]
Tobacco use screening	1,017	53	58	51	1.36 (1.003, 1.84) [.048]
Advising tobacco cessation	326	64	66	62	1.17 (0.68, 1.91) [.5]
Colon cancer screening	714	83	84	83	1.02 (0.63, 1.66) [.8]
Breast cancer screening	346	89	91	87	1.45 (0.68, 3.13) [.3]
Lipid screening	987	84	83	84	0.92 (0.60, 1.40) [.7]

^*P from univariate logistic regression, with generalized estimating equations to adjust for clustering of patients within providers.

PGY, postgraduate year; CI, confidence interval.

Colon cancer screening was a combined indicator including FOBT, flexible sigmoidoscopy, double-contrast barium enema, or colonoscopy. Although overall screening for colon cancer was not significantly different, patients seen by PGY-1 residents were more likely to receive FOBT compared with those seen by PGY-2 residents (40%, 30%, P=.009). After adjusting as above and further adjusting for receipt of other invasive colon cancer screenings, patients seen by PGY-1 residents were still more likely to receive FOBT (OR 1.46 (1.004 to 2.13), P=.048) compared with those seen by PGY-2 residents.

Note that when the analysis was stratified by the 2 academic years, no significant differences in PGY-1 and PGY-2 performance were seen for pneumococcal vaccination, tobacco cessation counseling, colon or breast cancer screening, or lipid screening. Again, PGY-1 residents tended to outperform PGY-2 residents for smoking screening (54% vs 45% for academic year 2000 to 2001 and 60% vs 53% for academic year 2002 to 2003).

DISCUSSION

Our results did not support the assumption that clinical performance improves with experience. We found that performance on 5 of the 6 preventive health services was no different when comparing PGY-2 and PGY-1 residents. In fact, patients seen by PGY-1 residents were significantly more likely to receive tobacco use screening than patients seen by PGY-2 residents, although the difference was small. Thus, overall performance was not superior in more advanced Internal Medicine residents.

Within the context of previous studies, our analysis is unique in that it focuses on specific indicators, and charts were abstracted before other quality improvement interventions were implemented. One prior longitudinal study demonstrated improved performance as residents progressed in their training from PGY-1 year to PGY-3 year, but these residents were involved in a chart audit and feedback program that was felt to be the cause of the improved performance over time.¹² Although limited by sample size, previous smaller studies of medical record audit and feedback have not demonstrated superior performance by senior residents compared with residents in their first year of training.^13,14 A larger study with over 6,000 patients demonstrated improved overall performance by primary care track residents over traditional residents, but no difference in year of residency training.¹⁵

There are several potential explanations for the findings of our study. First, at our institution, PGY-1 residents are given more time per patient encounter and have less patients scheduled per session compared to PGY-2 residents. This extra time may allow for increased attention to preventive health services. A study by Malone et al.¹⁶ demonstrated that in a residency clinic, the number of patients seen in the clinic session impacted the way in which residents allocated time. This study did not address the proportion of time spent in direct patient care activities, but one could postulate that the more patients a resident has to see in a given time period, the less time he or she has to address preventive health services.

Another explanation is that a “fresh patient effect” occurs, whereby one is more likely to address preventive health services when initiating a relationship with a patient. All patients seen by the PGY-1 residents are initially new patients to them. One can hypothesize that more attention is paid to previous preventive health services, smoking status, and assuring that the patient is up to date on screening tests during the initial visit. Once the patient returns for follow-up visits, a physician may be more likely to concentrate on managing the acute and chronic medical problems. In fact, previous studies by Ellerbeck et al. have shown that screening for colorectal cancer and smoking status were more common among new patients presenting to primary care clinics.^17,18

A third potential explanation for our findings is that attendings may provide closer supervision of PGY-1 residents, particularly in areas assumed to be appropriately managed by more senior residents. In the study by Malone et al.,¹⁶ PGY-3 residents had significantly less review time with the attending physician than the PGY-2 residents. In this study, staffing with an attending was left to the discretion of the resident, whereas in our clinic setting, all patient visits must be staffed with an attending.¹⁶ The intensity of the attending interaction and focus on preventive heath services has not been measured across year of residency training, but perhaps PGY-1 residents who are learning about preventive health services are receiving more teaching and, hence, show better performance in this area.

Another area of uncertainty is whether the lack of higher performance by PGY-2 residents is merely a reflection of better documentation by PGY-1 residents and not indicative of true performance. A prior study by Dresselhaus et al.¹⁹ demonstrated that resident physicians performed more preventive health services than they documented in the medical record. Chart abstraction underestimated the performance of preventive health measures for all 7 quality indicators measured in this study compared with self-reports by simulated patients and clinical vignettes designed to recreate the sequence of a typical patient visit.¹⁹ It is possible that PGY-2 residents are less comprehensive in documentation, and thus performance is underestimated, compared with PGY-1 residents.

Additionally, residents were given credit for the quality indicator if documentation of past performance was noted. The guideline-based appropriate time interval varies by indicator. For indicators not required on an annual basis, such as colon cancer screening and Pneumovax, measured performance could represent a cumulative effect of current and prior resident performance. This may have limited our ability to detect differences for some indicators. However, we feel that the current provider is responsible for reviewing prior preventive health services documentation and ensuring that their patients are up to date. Note that for those indicators with more narrow time intervals (tobacco use screening and FOBT), PGY-1 residents outperformed PGY-2 residents.

The major limitation of our study is that this represents our residency program's continuity clinic experience over a limited time period and may not accurately reflect the experience of other institutions or residency clinics. Our patients generally have low socioeconomic status and multiple complex medical problems that may make issues of preventive health services more difficult to address. In addition, we focused on quality indicators that we felt were relevant to residency education, could be accurately abstracted, had perceived variability among residents, and had potential for change. Thus, the 6 quality indicators we chose may not accurately reflect the performance of other preventive health services.

Also, we used 2 sets of cross-sectional data from PGY-1 and PGY-2 residents to assess the consistency of patterns over time. Certainly, unmeasured differences might exist between the 2 academic years that limit the generalizability of these results. Unmeasured differences might also be responsible for the difference seen in smoking cessation counseling. However, the measured resident characteristics and patterns of performance were fairly consistent across the 2 years. Also, we chose indicators that have remained constant over the designated time interval by national guidelines. Our attending population has also remained consistent since June 2000.

The primary strength of our study compared with prior studies is the large number of patient charts abstracted. We abstracted a total of 1,017 patient charts from 4 different classes of residents. Previous studies had patient chart samples ranging from 125 to 280.^3,15,20 This increase in sample size and power allowed more precision in the estimation of differences. We also focused on specific measures of preventive health services and not summary scores as in prior studies. This allowed us to assess differences across a range of indicators.

Another strength of our study is the quality of our data abstraction methodology. Allison et al.¹⁰ describe the complexities of obtaining high quality data by chart review. Our MedQuest software package, intense abstractor training, and careful construction of quality indicators minimize inappropriate interpretation of our results. We had a trained abstractor, blinded to the outcome of interest, with a standardized computer instrument with very low rates of error upon double abstraction. Other studies using chart audit by self, peers, or attendings, may lack reliability as prior medical training has been shown to interfere with abstraction quality.¹⁰

In conclusion, we found that PGY-2 residents did not show superior performance compared with PGY-1 residents across multiple preventive health services. Several reasons can be postulated to account for this difference, and these have important implications for residency education. Efforts should be made to increase awareness and attention to detail for preventive health services despite the limitations of shorter clinic appointment times and the “fresh patient effect.” Attendings should ensure an appropriate level of supervision for all levels of training and should not neglect questioning upper-level residents on the performance of preventive health services. Clinic systems should be developed to prompt preventive health services, and chart documentation should adequately reflect preventive health services offered, regardless of year of training. The assumption of the apprenticeship model of superior performance with advancing residency training may not be true in the outpatient continuity clinic. Specific efforts should be made not only to improve residents' outpatient clinical experiences, but to ensure quality patient care in the outpatient setting.