Assessing and Documenting the Cognitive Performance of Family Medicine Residents Practicing Outpatient Medicine

Abstract

Background

Development of cognitive skills for competent medical practice is a goal of residency education. Cognitive skills must be developed for many different clinical situations.

Innovation

We developed the Resident Cognitive Skills Documentation (CogDoc) as a method for capturing faculty members' real-time assessment of residents' cognitive performance while they precepted them in a family medicine office. The tool captures 3 dimensions of cognitive skills: medical knowledge, understanding, and its application. This article describes CogDoc development, our experience with its use, and its reliability and feasibility.

Methods

After development and pilot-testing, we introduced the CogDoc at a single training site, collecting all completed forms for 14 months to determine completion rate, competence development over time, consistency among preceptors, and resident use of the data.

Results

Thirty-eight faculty members completed 5021 CogDoc forms, documenting 29% of all patient visits by 33 residents. Competency was documented in all entrustable professional activities. Competence was statistically different among residents of different years of training for all 3 dimensions and progressively increased within all residency classes over time. Reliability scores were high: 0.9204 for the medical knowledge domain, 0.9405 for understanding, and 0.9414 for application. Almost every resident reported accessing the individual forms or summaries documenting their performance.

Conclusions

The CogDoc approach allows for ongoing assessment and documentation of resident competence, and, when compiled over time, depicts a comprehensive assessment of residents' cognitive development and ability to make decisions in ambulatory medicine. This approach meets criteria for an acceptable tool for assessing cognitive skills.

What was known

Developing residents' cognitive skills is an important goal of residency education.

What is new

The Resident Cognitive Skills Documentation (CogDoc) tool captures 3 dimensions of cognitive skills: medical knowledge, understanding, and its application.

Limitations

The tool was implemented in a single population of faculty preceptors and residents, limiting generalizability.

Bottom line

The CogDoc approach allows for ongoing assessment and documentation of resident competence.

Introduction

Developing cognitive skills is a major focus of residency. While medical students graduate with much knowledge, their critical thinking and decision-making skills are underdeveloped.¹ Thus, it is important to assess these developing skills over time in all areas of practice to ensure that graduates are competent.

Faculty can make these assessments during precepting. As a resident describes the data collected from patients and the thought process for arriving at a diagnosis and management plan, preceptors assess the resident's cognitive processing and decision-making in-situ.

Usually, this assessment of residents' critical thinking and decision-making skills is not documented (other than cosignature of the patient record). Instead, preceptors' general impressions are captured only during end-of-rotation summations and are typically not accurate.^2,3 By the end of a rotation, raters have difficulty recalling details, leading to simplistic conceptions of performance. In addition, positive feedback is reported more often, leading to overly generous assessments.² In addition, this summative approach may not identify specific weaknesses in resident performance. Decision-making skills are not generic, homogeneous, stable, inborn traits.^2–4 Excellent diagnostic and management skills are context-specific and expertise in 1 area of practice is not easily transferred, especially by novices, to other areas of practice.^5–9

Therefore, it is important to document cognitive performance in practice over time to assure that graduates are competent. What is needed is a method of capturing supervisors' expert assessments of resident capability in actual performance.¹⁰ These assessments will also document resident achievement of the Accreditation Council for Graduate Medical Education's educational Milestones.¹¹

To capture preceptors' assessments of residents' cognitive performance we developed the Resident Cognitive Skills Documentation (CogDoc) tool. It consists of 3 dimensions of performance—medical knowledge, understanding, and its application—with each having 3 levels of performance. A preceptor familiar with the scoring rubric can complete the form during or at the end of each patient presentation in about 10 seconds. In this article, we describe the development of this form, our experience with its use, its reliability, and residents' reactions to its use in resident assessment.

Methods

Assessment Form Development

The CogDoc approach reflects the concepts embodied in Bloom's Taxonomy of Cognitive Domains, revised.¹² The 3 dimensions (medical knowledge, understanding, and application) do not directly represent the same domains of Bloom's taxonomy but were descriptions that were intuitive in our pretesting with users. The medical knowledge scale encompasses Bloom's “remembering” and “understanding,” and documents a resident's determination of what information needs to be considered when making decisions for the specific patient. The understanding dimension (Bloom's “applying” and “analyzing”) documents how well residents solicit and synthesize information from their patients. The information gathering is based on medical knowledge and represents the residents' ability to create a coherent narrative on both medical knowledge and gathered patient information. The application dimension (Bloom's “evaluating” and “creating”) assesses how the resident builds on the understanding of the patient, and his or her knowledge of medicine to create an appropriate, individualized course of action.

Residents are assessed by using criteria describing the expected competent performance for 1 of 60 outpatient entrustable professional activities (EPAs).¹³ The activities encompassing each EPA are outlined on a separate linked form available to preceptors and residents (box).

box Example of an Entrustable Professional Activity Description

EPA: Managing the Patient With Symptoms of Dyspepsia Diagnosis/diagnostic activities

• Demonstrate knowledge of alarm signs (red flags) and indications for endoscopy

• Contrast presenting symptoms of gastroesophageal reflux disease (GERD), peptic ulcer disease, and gastritis

• Describe when Helicobacter pylori testing is necessary and describe the advantages and disadvantages of testing methods

• Know when screening for Barrett esophagus is required

Treatment activities

• Show or know a rational treatment algorithm

• Show or know counseling about lifestyle changes to decrease dyspepsia and GERD symptoms

• Know the relative costs of treatment

In addition to these 2 categories of activities, EPA descriptions may also include primary prevention activities, secondary prevention and monitoring activities, knowledge and prevention of secondary morbidities, and quality improvement and systems-related activities.

For each dimension, faculty can rate cognitive performance to be below, at, or above competence. The form provides criteria for each level of competence, which are standardized (criterion referenced) and not based on a “typical” resident at the same level of training (norm-referencing). A free-text box on each form labelled “identified learning needs” allows faculty members to provide residents with further feedback.

Study Design

We conducted this study over a 14-month period in a single family medicine office in which 24 to 25 residents per year see their own outpatients. Data were collected for 33 residents.

We introduced a paper version of the CogDoc form in October 2008 after initial pilot testing. In April 2010, we switched to a computerized version. At a faculty development retreat, we explained the background and purpose to our core faculty members and showed them how to complete it using role-plays of precepting sessions to illustrate the scoring.

We introduced residents to CogDoc in a series of meetings, emphasizing that it is designed to provide ongoing documentation of their performance and is not a series of “tests.” The chief residents championed the acceptance of CogDoc among the residents.

Preceptors complete a CogDoc form for every patient encounter discussed by a resident, either during or after the precepting encounter. The preceptors select a level of demonstrated competence on each of the 3 dimensions (with the option of not completing a dimension). Each encounter is linked to 1 of 60 outpatient EPAs¹³ that best reflect the nature of that patient visit. In this way, completed CogDoc scores can be organized by EPA.

We surveyed residents to determine their use of CogDoc reports. We conducted the anonymous survey by using an online survey tool (SurveyMonkey).

The study was approved by the Institutional Review Board of the Cambridge Health Alliance.

Analysis

Data from every CogDoc form completed between the introduction of the form on April 28, 2010, through June 30, 2011, were downloaded to a spreadsheet for analysis. Results were coded by resident being assessed, faculty member assessor, and date.

Internal consistency for each dimension on the form for individual residents was used to evaluate the reliability of CogDoc. We selected completed CogDoc forms for third-year residents for their top 5 most common diagnoses. We assumed that, for these diagnoses and at this stage of training, each resident would demonstrate consistent performance most of the time and therefore most of the difference in CogDoc scores would be due to differences among raters. Cronbach α was used to compare CogDoc scores for each resident.

Results

A total of 5021 CogDoc forms were completed by 38 faculty members for 33 residents (during 1.3 academic calendar years). Competency was documented in all 60 EPAs (results reported separately¹³), though no individual resident had completed competency categories documented for all 60 EPAs during the 14-month period (range for second- and third-year residents, 68%–95%).

A total of 29% (5021 of 17 304) of all patient visits conducted by residents were documented via CogDoc (range per month, 16%–55%). Documentation of visits varied by postgraduate year (PGY) of training: 27% (2419 of 9058) of PGY-3 visits, 30% (2000 of 6639) of PGY-2 visits, and 38% (602 of 1607) of PGY-1 visits.

For academic year 2010–2011, we compared results on each dimension by class year and by academic quarter to determine whether competence documentation was different among the class years and whether they changed over time (figures 1 through 3). Residents were rated as “competent” if their performance was documented as “competent” or “above competent” on each of the dimensions.

FIGURE 1.

Percentage of Visits Rated Competent (2) or Above Competent (3) in “Medical Knowledge,” by Year of Training

FIGURE 2.

Percentage of Visits Rated Competent (2) or Above Competent (3) in “Understanding,” by Year of Training

FIGURE 3.

Percentage of Visits Rated Competent (2) or Above Competent (3) in “Application to Patient,” by Year of Training

To assess reliability, we used completed forms for all third-year residents for the 5 most common EPAs for these residents (n  =  675), which included a patient with joint pain, a patient with, or at risk for, cardiovascular disease, a patient with a lesion or a rash, a pregnant patient, and a patient with diabetes. These EPAs compose 28% of CogDocs completed for these residents. Internal consistency (Cronbach α) was 0.92 for the medical knowledge domain, and 0.94 for understanding application.

Almost all residents (97%, 32 of 33) responded to the survey asking about their response to and use of the feedback. Most residents (91%, 29 of 32) reported looking at completed CogDoc forms and 97% (31 of 32) reported looking at the graphic summary. Most respondents (59%, 19 of 32) reported reading most of the forms completed for them, while 35% (11 of 32) reported that they usually looked only at forms documenting below competence on 1 or more of the scales. Almost all respondents (82%, 26 of 32) agreed the CogDoc assessments usually reflected their performance.

Discussion

Using a computer-based approach to collecting and summarizing the documentation, we developed CogDoc, a tool that was easily implemented and provided reliable formative assessment used by most residents.

Two other approaches to assessing clinical decision making have also been developed. Precept-Assist is a computer evaluation system that documents residents' performance, based on preceptors' summative assessment, for the complete patient session.¹⁴ The “Real-Time Evaluation of Doctors' Independence” assessment method tracks resident performance for individual patients during precepting.¹⁵ It does not identify specific strengths and weaknesses.

Our assessment approach has limitations. Assessments represent expert judgment based on what a resident reported about the patient visit, not direct observation of the resident with the patient. In addition, since the primary reason for precepting is to assure good patient care and to improve resident knowledge, not to provide an ideal assessment, preceptors may not ask enough questions to completely assess a resident's cognitive skill.

In addition, only 1 in 4 patient visits was documented by using the forms. This rate is about the same as the 30% reported by Reichard.¹⁵ Based on informal discussions with preceptors, it is possible that cases documented are bimodal, representing the best and worst performance observed. In addition, scores on the 3 scales often were similar on a specific form, which might reflect a “halo” effect, the tendency to rate a person high on all scales because of a global impression. The CogDoc tool has only been studied in a single population of faculty preceptors and residents. Further research on its implementation and use in other family medicine residency settings is necessary for confidence in its reliability.

Determining the reliability of this method is difficult using standard education metrics.^16,17 The ratings are in part dependent on the interaction between the resident and the preceptor; the questions asked by the preceptor will allow greater or lesser demonstration of knowledge, understanding, and application by the resident. As a result, it is not reasonable to assess interrater reliability by using video recording of resident-preceptor discussions or even comparing ratings of 2 preceptors assessing a single resident at the same time.

Our attempt to assess interrater reliability is further confounded by the issues of intercase reliability, a problem that plagues competence assessment.¹⁶ The value of this method of assessment is that it is inexpensive, fast, and captures existing data (eg, the preceptors' assessments of resident decision-making performance). A strength of our approach is the 30% rate of clinical encounters being assessed, as a major contributor to reliability is the frequency of the assessments.¹⁸ Finally, assessments focused on EPAs is 1 of several assessment methods that can be combined to support Milestone determination required of the Next Accreditation System.¹⁹