Incorporating Performance Improvement Methods into a Needs Assessment: Experience with a Nutrition and Exercise Curriculum

Abstract

BACKGROUND

Clinical guidelines recommend that physicians counsel patients on diet and exercise; however, physician counseling remains suboptimal.

OBJECTIVES

To determine if incorporating performance improvement (PI) methodologies into a needs assessment for an internal medicine (IM) residency curriculum on nutrition and exercise counseling was feasible and enhanced our understanding of the curricular needs.

DESIGN AND PARTICIPANTS

One hundred and fifty-eight IM residents completed a questionnaire to assess their knowledge, attitudes, and practices (KAP) about nutrition and exercise counseling for hypertensive patients. Residents’ baseline nutrition and exercise counseling rates were also obtained using chart abstraction. Fishbone diagrams were created by the residents to delineate perceived barriers to diet and exercise counseling.

MAIN MEASURES

The KAP questionnaire was analyzed using descriptive statistics. Chart abstraction data was plotted on run charts and average counseling rates were calculated. Pareto charts were developed from the fishbone diagrams depicting the number of times each barrier was reported.

KEY RESULTS

Almost 90% of the residents reported counseling their hypertensive patients about diet and exercise more than 20% of the time on the KAP questionnaire. In contrast, chart abstraction revealed average counseling rates of 3% and 4% for nutrition and exercise, respectively. The KAP questionnaire exposed a clinical knowledge deficit, lack of familiarity with the national guidelines, and low self-efficacy. In contrast, the fishbone analysis highlighted patient apathy, patient co-morbidities, and time pressure as the major perceived barriers.

CONCLUSIONS

We found that incorporating PI methods into a needs assessment for an IM residency curriculum on nutrition and exercise counseling for patients at risk of cardiovascular disease was feasible, provided additional information not obtained through other means, and provided the opportunity to pilot the use of PI techniques as an educational strategy and means of measuring outcomes. Our findings suggest that utilization of PI principles provides a useful framework for developing and implementing a medical education curriculum and measuring its effectiveness.

INTRODUCTION

For years, the Joint National Committee on Prevention, Detection, Evaluation, and Treatment of High Blood Pressure (JNC) and the American Heart Association (AHA) have recommended that physicians help patients at risk for cardiovascular disease (CVD) set dietary and physical activity goals to help control blood pressure and reduce their CVD risk.^1,2 Regular aerobic activity of at least 30 minutes has been shown to decrease systolic blood pressure (BP) by 4–9 millimeters of mercury (mmHg), and the Dietary Approaches to Stop Hypertension (DASH)-low sodium diet can lower systolic BP 11.5 mmHg,¹ an effect similar to single drug therapy. The 2009 report of the American College of Cardiology Foundation and American Heart Association (ACCF/AHA) on primary prevention of cardiovascular disease included both “dietary intake counseling” and “physical activity counseling” as key performance measures for all adult patients, regardless of reason for visit.³ Despite both clinical evidence and national recommendations, physician counseling on exercise and diet remains suboptimal.⁴

The J. Willis Hurst Internal Medicine Residency program at Emory University School of Medicine is a large training program with 170 residents who are graduates of medical schools from across the country. Ninety percent of the residents have their continuity clinic at Grady Memorial Hospital. The primary care clinics at Grady Memorial Hospital serve a population that is largely older, African American with lower literacy, lower socioeconomic status, and higher co-morbid conditions. Diet and exercise counseling is particularly relevant given the high overall cardiovascular risk in this population (approximately 78% of the patients are hypertensive, 45% obese, 38% hypercholesterolemic, 26% diabetic, 23% smokers, and 39% sedentary).⁵ Further, low counseling rates for cardiovascular disease prevention have previously been documented in this population.⁶

With the intent of improving low exercise and nutrition counseling rates among residents in the continuity clinic setting, we used the framework of Kern’s six-step approach to curriculum development for medical education,⁷ where the first two steps are to perform a problem identification and general needs assessment followed by a targeted needs assessment. We developed and describe here an innovative approach to performing a curriculum needs assessment, incorporating performance improvement (PI) tools⁸ into a standard literature review and survey approach.

OBJECTIVES

Our objective was to incorporate PI methodologies into a needs assessment for a curriculum that addresses residents’ low counseling on diet and exercise in hypertensive patients. Specifically, we set out to determine if 1) introducing the residents to PI concepts and tools during the needs assessment was feasible and if 2) use of PI tools enhanced our understanding of curriculum needs above that obtained through use of traditional literature review and survey techniques.

METHODS

We performed a comprehensive needs assessment based on Kern’s six-step approach. For step one, Kern describes a generalized needs assessment as “the identification and critical analysis of a health care need.” We reviewed clinical practice guidelines on physical activity and nutrition counseling issued by the JNC,¹ the ACCF/AHA,³ and the American College of Sports Medicine (ACSM).^9,10 We placed these recommendations in the context of competency-based residency training per the Accreditation Council of Graduate Medical Education (ACGME) and the American Board of Internal Medicine (ABIM), looking at curriculum development in physical activity and nutrition counseling as it relates to the six core competencies.¹¹ Literature review confirmed low counseling rates of patients at risk for CVD in our program⁶ and in other residency programs^12–14 and documented studies on reported barriers to physician counseling.^15–17

We then conducted a targeted needs assessment defined in step two of Kern’s six-step approach as “assessing the needs of one’s targeted group of learners and their medical institution/learning environment, which may be different from the needs of learners and medical institutions in general.” At study initiation in 2006, faculty preceptors perceived nutrition and exercise counseling rates remained low in resident clinics. We convened a working group composed of residency and clinic leaders, key faculty, and a medical chief resident, which utilized several strategies to assess learner needs:

1. A resident questionnaire on medical knowledge, professional attitudes, and patient care practices (KAP) related to diet and exercise counseling in continuity clinic patients with hypertension

2. Creation of a run chart (Table 1) from chart abstraction of resident documentation of diet and exercise counseling

3. Creation of a fishbone or cause and effect diagram (Table 1) by the residents

4. Creation of a Pareto chart (Table 1) depicting the frequency of barriers the residents identified during their fishbone session

Table 1.

Definition of Terms

Term	Definition
Run Chart	Employed to visually represent data over time. Points are plotted on a graph and connected in the order in which they are available. The x-axis represents time (e.g. days, weeks, months) and the y-axis represents the measurement of interest (e.g. frequency of counseling) at that point in time. An average is calculated and graphically represented as a horizontal line. A run chart allows a process to be monitored to see if the long-range average is changing. See Figure 2
Statistical Process Control (SPC) Chart	A SPC chart is a run chart that shows data over time with statistically determined upper and lower control limits depicted as lines drawn horizontally above and below the process average. A control chart depicts the degree of variation in a process over time. It enables you to visually track data following implementing an intervention and determines whether variability seen with your data is due to random variation or non-random/your intervention variation
Fishbone/Cause & Effect Diagram	Diagram that allows a team to structure a brainstorming session and graphically display the relationship between some “effect” or problem and all of its possible “causes.” The diagram resembles the skeleton of a fish. The problem at hand (e.g. low counseling rates) is placed at the head of the fish. Potential causes of the problem are grouped into major categories and form bones of the fish. See Figure 3
Pareto Chart	Chart that displays the relative frequency of causes of a problem in a descending bar graph. Each cause is listed on the x-axis and the y-axis is a measure of frequency, totaling 100%. It is based on the Pareto principle: 20% of the sources cause 80% of any problem, and helps a team focus on causes that will have the greatest impact if solved. See Figures 4 & 5

We describe each of these strategies in detail below.

KAP Questionnaire

The working group convened regularly to develop a resident survey, as no validated instrument existed in the literature. Content experts developed and pilot tested the KAP questionnaire. A literature search provided the conceptual framework for important content addressed in the questionnaire. We based our medical knowledge questions on the JNC 7¹ guidelines for lifestyle counseling and physical activity. We selected the DASH—low sodium diet and regular aerobic exercise as the focus of our knowledge questions. To assess professional attitudes toward counseling, we developed open and closed ended questions regarding potential barriers based on published barriers to counseling.^12–17 We then developed questions on resident practices in physical activity and nutrition counseling. The self-administered KAP survey used multiple choice questions for the knowledge and practices portions and 5-point Likert scale questions for the attitudes portion. After pilot testing and revision to ensure face validity, the final questionnaire was administered to all residents, who were required to complete it, in an observed setting over a 2-week period.

Data were entered into Microsoft Excel, using double data entry to reduce errors. Data were then imported into SAS v9.2 (Cary, NC) in order to generate frequencies and descriptive statistics.

Chart Abstraction

Retrospective chart abstraction from randomly selected charts established nutrition and exercise counseling rates for the residency cohort. We used a strategy of random chart selection to obtain group performance measures to compare to the KAP survey results. The investigators created a chart abstraction form which was pilot tested prior to active data collection to consistently abstract data. Abstracted data included: 1) presence of hypertension on the patient’s problem list or assessment note; 2) patient’s blood pressure at time of visit; and 3) documentation of exercise or nutrition counseling. While ultimately striving to improve counseling using the JNC 7 recommendations, in our baseline chart abstraction we gave credit if there was any mention of: a) counseling consistent with a DASH-low sodium diet (fruits, vegetables, low fat dairy products, low saturated and total fats, or sodium reduction) but not for caloric restriction alone, and b) counseling on frequency or duration of exercise or any recommendation regarding an exercise plan.

For our baseline chart abstraction, we collected data to create a run chart (Table 1) which would allow us to visualize the trend in counseling over time and calculate an average counseling rate. However, our chart abstraction strategy was designed with the intent of eventually creating a statistical process control (SPC) chart (Table 1). The SPC chart is a type of run chart but requires more data points (typically 20–25) for calculation of upper/lower control limits. An SPC chart would allow us to determine whether variations seen in our counseling rates as we implemented our curriculum were due to random variation or to a change in behavior, possibly, because of our curriculum.

To create our run chart, we collected data for one week, 6 times, over a period of 6 months. Each data collection week, twenty charts were randomly selected using the following strategy: patient visits for resident clinics were numbered sequentially and charts were selected by medical record clerks using a random number table. Charts were included for abstraction if the diagnosis of hypertension was listed on the clinic chart problem list or in the clinic note assessment. The investigators performed the chart abstraction. The percent of visits in which nutrition and exercise counseling was documented was calculated from the weekly chart abstraction and used to create the following run charts: 1) Documented Nutrition Counseling and 2) Documented Exercise Counseling. Average counseling rates for the 6-month period were calculated from this data.

Fishbone Diagram

The residents generated barriers to conducting nutrition and exercise counseling in hypertensive patients by creating a fishbone diagram (Table 1) during a one-hour continuity clinic conference. The faculty divided the residents into groups of 10–12. Half of the groups focused on barriers to exercise counseling and the other half focused on barriers to nutritional counseling. Each resident was given an adhesive notepad and asked to silently spend 5–10 minutes writing down as many barriers as possible, one barrier per piece of paper, to counseling. Then, the groups worked together for 10 minutes organizing related barriers into categories of their choice. The residents then drew a fishbone diagram on a 25 inch × 30 inch sheet of paper, placing the statement, “Barriers to Exercise (Nutrition) Counseling in Patients with Hypertension in Clinic” on the head of the fish. The residents then placed the adhesive notes grouped into categories onto the bones of the fish. The residents were instructed to put all the adhesive notes, including those that had the same barrier listed multiple times (by different residents) on the diagram to ensure that we would be able to later count how often each barrier was mentioned. By following this process, rather than simply brainstorming in groups, the residents used a structured approach that allowed everyone to give input, provided a visual display of each team’s thoughts, and ultimately stimulated a more thorough exploration of the problem.

Pareto Chart

The investigators created an aggregate fishbone diagram for exercise or nutrition by combining the findings from all the sessions. These aggregate fishbone diagrams were used to make Pareto charts (Table 1). The number of times a barrier was identified by a resident was depicted in the Pareto chart as the frequency said barrier was identified.

RESULTS

KAP Survey

All 158 of the Emory IM residents who had their continuity clinic at Grady Memorial Hospital completed the KAP questionnaire. There were 57 (37%) post-graduate year (PGY) 1, 48 (31%) PGY 2 and 49 (32%) PGY three residents (four residents did not identify their PGY) who completed the questionnaire. Figure 1 summarizes the resident’s self-reported counseling behaviors. Roughly, 90% of the residents reported counseling on diet and exercise in greater than 20% of their visits with their hypertensive patients.

Figure 1.

IM residents’ nutrition and exercise counseling rates: self reported rates (bars) and chart abstraction rates (lines).

Perceived barriers to counseling were examined. The minority of the residents strongly agreed that insufficient time or lack of patient interest were barriers to counseling hypertensive patients on diet (n = 25, 15.9% and n = 7, 4.5%, respectively) and exercise (n = 15, 9.7% and n = 12, 7.7%, respectively). Lack of knowledge and comfort with dietary and exercise counseling appeared to be a major barrier to counseling. Only three (1.9%) residents strongly agreed that they felt prepared to follow evidence-based guidelines on diet counseling. Similarly only nine (5.8%) residents strongly agreed that they felt prepared to prescribe exercise to patients. Additionally only one (0.7%) resident strongly agreed that he felt successful or confident in his ability to improve hypertensive patients’ diets and only three (2.0%) residents strongly agreed that they felt successful or confident in their ability to get their hypertensive patients to start exercising. Case-based questions to assess residents’ knowledge confirmed a knowledge deficit of the evidence-based guidelines for nutrition and exercise counseling in patients with hypertension.

Resident attitudes towards counseling were also examined. Most residents strongly agreed that all hypertensive patients should be counseled on diet (n = 117, 74.5%) and exercise (n = 78, 50.3%) but less than half the residents believed that patients should be counseled at every visit on diet (n = 46, 29.5%) and exercise (n = 37, 23.6%). Seventy-nine (51.3%) of the residents strongly agreed that it was their responsibility as a physician to counsel hypertensive patients on diet but only 57 (36.5%) strongly agreed about this for exercise. Note that the results presented here and from the other needs assessment methods are aggregated results.

Chart Abstraction

One hundred and three charts were selected. Six of those charts were excluded because they were not resident charts and three charts were excluded due to illegibility. Data were abstracted by the investigators from 94 charts. The average counseling rate for each data collection week was calculated and plotted on run charts depicting documented counseling for patients with hypertension on exercise and nutrition over time. The run charts both showed low rates of counseling ranging from 0–15%. Figure 2 is the run chart for documented nutrition counseling. The average nutrition and exercise counseling rates of 3% and 4%, respectively was much lower than that reported by the residents on the KAP as depicted in Figure 1.

Figure 2.

Run chart of IM residents’ documented nutrition counseling.

Fishbone Diagrams and Pareto Charts

Figure 3 is the aggregate fishbone diagram for nutrition counseling. The Pareto charts in Figures 4 and 5 detail all the residents’ perceived barriers. The residents identified patient co-morbidities, patient apathy, lack of time, patient lack of resources, and lack of physician knowledge, as the major barriers to counseling hypertensive patients on diet. They identified patient apathy, patient co-morbidities, lack of time, and patient lack of resources as the major barriers to counseling hypertensive patients on exercise.

Figure 3.

Aggregate fishbone for IM residents’ perceived barriers to nutrition counseling in hypertensive patients.

Figure 4.

Pareto chart of IM residents’ barriers to nutrition counseling.

Figure 5.

Pareto chart of IM residents’ barriers to exercise counseling.

DISCUSSION

As we move into the era of competency-based medical education, a systematic approach to curriculum planning is vital. Information obtained through curricular needs assessment can help guide curriculum development and selection of outcomes measures. We used an innovative approach to conduct a nutrition and exercise counseling curriculum needs assessment, combining literature review, self-administered resident survey, and PI methods. The PI method of creating fishbone diagrams uncovered perceived barriers to counseling not exposed by our resident questionnaire on counseling knowledge, attitudes, and practices. Use of chart abstraction to measure frequency of counseling highlighted very low counseling rates and underscored for the residents the need to document when counseling does occur.

As outlined in Kern’s approach to curriculum development, targeted assessment of learners’ needs facilitates development of more specific curricular goals and objectives. Reliance on a traditional needs assessment technique of surveying our learners would have led us to focus on addressing the clinical knowledge deficit, lack of familiarity with the national guidelines, and low self-efficacy, which the survey revealed as the major barriers to counseling patients. However, adding the fishbone analysis demonstrated that our curricular goals and objectives also needed to address resident perceptions that patient apathy, patient co-morbidities, and time pressure were major barriers to dietary and exercise counseling. In this regard, the fishbone findings were in contrast with the KAP findings, where the minority of the residents strongly agreed that insufficient time or lack of patient interest were barriers to counseling hypertensive patients on diet and exercise.

We also demonstrated the feasibility and value of utilizing PI methods in addition to our KAP survey in our needs assessment. Creating the fishbone diagrams directly engaged residents in brainstorming on barriers to counseling. The fishbone diagrams provided a framework for graphically displaying barriers as a Pareto diagram, highlighting the most frequently cited barriers to counseling. These PI methods allowed us to not only identify deficiencies, but to focus our curriculum on areas with the greatest potential impact. We felt that an educational strategy where resident teams developed PI projects to address identified barriers would be a natural progression from barrier identification to implementation of potential solutions. Our experience during this needs assessment also allowed us to consider the resources (personnel, space, and time) necessary to continue integrating PI methods into the next steps of our curriculum implementation and evaluation.

The discrepancy between resident self-reported counseling and documented counseling reinforced the importance of using chart abstraction to create run charts as a strategy to evaluate the outcome of our curriculum. As in other studies, our study’s self-reported frequency of counseling was higher than that found by chart abstraction. The true counseling rate likely lies between that found by survey and chart abstraction but utilizing the strategy of sampling resident charts over time and creation of a run chart to monitor the outcome of our curriculum has several attractive features. Chart documentation is a more stringent criterion for determining counseling rates and eliminates recall bias. Additionally, continued collection of counseling data over time allows us to use our run chart to create a SPC Chart. This allows us to determine whether variations seen in counseling rates as we implemented our curriculum were due to random variation or a statistically significant change in rates due to resident-initiated interventions to improve counseling.

There are limitations to this study. First, although our residents represent medical schools from across the country, this study sample draws from IM residents at a single institutional site and it may be difficult to generalize to residents at other institutions. Secondly, although we used a random number chart to choose charts for abstraction, PGY 2 and PGY 3 residents were over-represented due to their larger patient panels. Third, although the investigators used a standard data collection instrument with stringent rules we may not have been able to eliminate all the bias inherent in the investigators performing the chart abstraction or in creating the aggregate fishbone diagrams and Pareto charts. Additionally, for the fishbone analysis, the residents identified their perceived barriers to counseling and therefore we may not have captured all the important barriers to counseling the patients, and some of the identified barriers may not be true barriers (e.g. patient apathy). Finally, barriers to counseling patients identified during residency may not reflect all the challenges that practicing physicians face, such as lack of reimbursement.

CONCLUSION

In conducting a needs assessment for our IM residency curriculum on exercise and nutrition counseling, we used an innovative approach that incorporated PI methodology with literature review and survey techniques. We found that use of PI methods was both feasible and provided additional information not obtained through other means. As medical education moves away from its traditional focus on measuring knowledge using test scores toward performance assessment, integration of PI principles offers a useful framework in developing and implementing a curriculum and measuring its effectiveness.