Physician Practice Behavior and Practice Guidelines: Using Unannounced Standardized Patients to Gather Data

Abstract

Background

Measuring actual practice behaviors of physicians, particularly as they relate to established clinical guidelines, is challenging. Standardized patients provide one method of collecting such data.

Objective

To demonstrate the use of unannounced standardized patients in gathering data that may address adherence to guidelines in an office setting.

Design

Unannounced standardized patients (SPs) simulating an initial type 2 diabetic visit presented to community offices of 32 internists as “real” patients to record physicians’ evaluation and management.

Participants

Unannounced SPs presented to the office of 32 internists as “real” patients.

Measurements

Unannounced SPs, simulating type 2 diabetics, completed a standardized assessment sheet, based on ADA guidelines to record physicians’ evaluation and management following an initial visit. Patient charts were also reviewed to determine if evaluation adhered to the guidelines.

Results

Unannounced SPs recorded 56 visits with 32 community internists; all SPs remained undetected. All physicians asked SPs about medications. At least 50% of physicians asked about home blood sugar monitoring, last eye exam, smoking, edema, and told patients to stop smoking. Less than 50% of physicians asked about parasthesias, performed fundoscopy, examined feet, referred the patient to a diabetic educator or ophthalmologist, or gave patients suggestions regarding glucose monitoring or exercise. HbA1c was ordered in 78%, metabolic profiles in 86%, and urinalysis/microalbumin in 41% of patients.

Conclusions

Unannounced standardized patients can successfully collect important data regarding physician practices in community settings. This method may be helpful in assessing physician adherence to established clinical practice guidelines.

INTRODUCTION

The use of standardized patients (SPs) to assess clinical skills and behaviors has become a routine practice in the evaluation of medical students^1,2. As lay people trained to simulate patients with real disorders, SPs can present in a consistent, reproducible manner. They can also be trained to effectively document and record participants’ interpersonal and clinical skills following each encounter^3,4. The value of SPs in assessing professional performance of practicing physicians has also been demonstrated, but their use in evaluating adherence to guidelines for the management of common clinical problems by direct physician observation is limited^5–10. To assess real practice behaviors of physicians, one would have to insert undetected (unannounced) standardized patients directly into practice settings. Few such reports are available^10–15, and most have focused on preventive care for otherwise asymptomatic adults. The use of “unannounced” standardized patients provides a unique opportunity to observe physician behavior in adhering to standard clinical practice guidelines for common chronic conditions such as diabetes.

The American Diabetes Association (ADA) publishes clear evidence-based goals for metabolic control and guidelines for patient management¹⁶. Yet previous studies have shown that adherence to these guidelines and achievement of metabolic goals in practice are suboptimal^17–20. Collection of data is generally limited to retrospective chart reviews or insurance claims data. To avoid these limitations, unannounced standardized patients, each representing a “typical” middle-aged male patient with type 2 diabetes, were used to assess physician behaviors of community-based internists as part of an educational research project. This study provides a unique look into the actual practice behaviors and adherence to ADA clinical practice guidelines of internists as they evaluated the first visit of a previously diagnosed uncomplicated type 2 diabetic.

METHODS

As part of an educational research project focused on training students and residents in non-academic, community-based ambulatory settings, internists in the greater New Orleans area were contacted and asked to participate and serve as preceptors. Internists who either expressed an interest or were recommended by peers agreed to enroll as either a participant or served as controls. All preceptors were subsequently assigned students and residents in their offices, while control physicians had no trainees. Sixteen physicians were enrolled over 2 consecutive years. Each year, eight of the internists served as participants and had the same student/resident team assigned to their office 1 afternoon every 1 to 2 weeks, while eight internists served as controls and had no trainees in their office. In year 1 of the project, nine of the physicians were recruited from one large multi-specialty group, two from other multi-specialty practices, and the remaining physicians were in private practices. Because of the logistics in dealing with multiple sites, in year 2, all physicians were recruited from the same large multi-specialty practice that participated in year 1. While approximately half of the recruited physicians were sub-specialists, all their practices included a substantial proportion of general internal medicine.

The study was submitted to the Institutional Review Board, which determined that informed consent was not necessary. However, all participating community physicians did sign a written agreement permitting the placement of SPs into their normal office schedules exactly as if they were “real” patients. To assure there would be no detection of SPs, a pseudo-healthcare plan (HMO) was created, from which all SP-related activity was coordinated. To prevent office personnel from inadvertently disclosing that a patient was an SP or that the HMO did not truly exist, information regarding the SPs was limited to one senior office manager at each practice location. Additionally, a letter was generated for other office personnel that indicated that the pseudo-HMO had been added as an accepted insurer several months in advance of scheduling any standardized patients. Once this information was provided, a coordinator from the standardized patient program began to schedule SP visits with the community physician offices. Each SP was given a pseudo-insurance card that requested that all visits be approved by calling a telephone number that went directly to the research project coordinator. When these patients presented to the physician offices, they registered by showing their pseudo-insurance card, and when approval was requested, the project coordinator, posing as the health-care provider employee, granted it. A post office box was obtained and listed as the HMO billing address for each SP. A flat fee payment was established and agreed to in advance for each SP visit to compensate for the loss of revenue caused by using a patient slot on the physician schedule.

The SPs entered each practice setting in an identical manner to existing patients, complete with the creation of a new patient office chart. Patients were seen and examined by either study or control preceptors in a manner no different than “real” patients. The original study design anticipated being able to evaluate the participating preceptors before and after they had student/resident teams in their offices, but making appointments at appropriately designated times proved logistically very difficult. Therefore, almost all SP visits for both participating and control preceptors were completed without the presence of trainees. If laboratory studies were requested, the registration card specifically requested that patients be sent to a large, commonly used reference laboratory. If other diagnostic studies were requested (radiographs, etc.), SPs asked to complete them at later dates. Similarly, if previous medical records were sought, the SP reported to the physician that he did not have the prior doctor’s name or address. All visits were planned as one-time only visits.

To assess the community physicians’ clinical behaviors, a standardized scenario was created and replicated by each SP at all of the office visits. A total of 15 SPs were trained to present as a middle-age, long-standing, type 2 diabetic male who had recently moved to the area and needed to establish a new health-care provider. Each “patient” was asymptomatic, but gave a history of type 2 diabetes and hypertension. SPs indicated that they smoked, did not monitor glucose, and did not exercise or watch their diets. All patients indicated that they were taking only metoprolol for hypertension and glipizide for their diabetes. All SPs were trained in advance to observe physician history-taking and physical examination behaviors included on a 64-item standardized assessment sheet, based on published clinical practice guidelines. Because the same assessment was originally intended also to evaluate students and included more basic aspects of the exam, not all 64 items were utilized in the final analysis of the practicing physicians. The checklist focused on items demonstrating that physicians sought evidence of preventive care and diabetic end-organ damage. Physicians were credited for any behaviors when they were addressed in any part of the history, or if it was possible that a physical exam maneuver was attempted. For example, physicians were credited with performing fundoscopy whenever an ophthalmoscope was used, while removal of socks was necessary for examination of the feet. Following each visit SPs were de-briefed about their experience by the SP trainer to maximize the completeness for each checklist. Inter-rater reliability of SPs was measured by having non-participating medicine residents complete scenarios, under the observation of the study co-investigators.

Office charges and diagnostic studies were also recorded. Following the completion of the research project, each patient chart was retrieved from the physician offices by the office manager, and the physician records were reviewed to determine what patient education and instructions were given, and what diagnostic studies were ordered. The cost of the visit, based on the Evaluation and Management level of service, and the cost of the diagnostic studies that were ordered were also determined. As a condition of participating, physician offices requested information regarding the systems and personnel utilized in registering new patients based on comprehensive notes on waiting times, attitudes and professionalism of the office and nursing staff observed by the SPs.

Data on all preceptor visits over a 2-year period are included in the present report. Due to complexities in the timing of these SP visits in relation to when physicians worked with a student/resident team, it was not possible, as originally planned, to evaluate the effect of precepting trainees on physician performance. Therefore, all physician data collected were pooled for evaluation. The frequency of physician behaviors, including adherence to published clinical practice guidelines while performing a first visit for a patient with uncomplicated type 2 diabetes, is presented with simple descriptive statistics.

RESULTS

Over 2 years, unannounced SPs scheduled appointments with 32 community internists. There were 56 SP visits, with no SP ever detected as not being a “real” patient. All visits were scheduled when learners were not expected to be present, though in several visits, a student did observe the preceptor. The maximum number of SP visits for any one physician was 2. Neither the presence of a student nor seeing another SP appeared to affect physician behaviors. Tables 1 and 2 present the data summarizing physician performance and the percent of physicians who adhered to the ADA guidelines for a comprehensive evaluation. Recommended features of the history and physical examination, laboratory studies, referrals, and patient education are shown. The items evaluated represent most components of the ADA guidelines on a comprehensive evaluation of a patient with diabetes. Many features of the history or examination were performed by less than half of the physicians, with less than 10% performing a sensory examination of the feet. There was also significant variation both among physicians and from the ADA guidelines for the laboratory studies that were ordered at this “first visit.” This was also demonstrated in the total charges for visits, including laboratory studies, which ranged from $150 to over $800.

Table 1.

Features of the History and Physical Examination and Percent of Physicians Performing Recommended ADA Guidelines

Features of the medical history asked by physicians:	Percent of physicians who performed the task:	Features of the physical examination performed by physicians:	Percent of physicians who performed the task:
Symptoms at diagnosis	35.7%	Blood pressure	62.5%
Glucose monitoring	67.9%	Fundoscopic examination	34%
Symptoms of hypoglycemia	28.6%	Examination of carotid and peripheral pulses	75%
Symptoms of hyperglycemia	41.1%	Thyroid palpation	58.9%
Current medications	100%	Feet	32.1%
Medication history	48.2%	Sensory exam of feet	9%
Eye complications	60.7%
Last eye exam	50.0%
Chest pain/tightness	82.1%
Previous MI	53.6%
Parasthesias in feet	42.9%
Leg pain with exercise	23.2%
Hypertension	42.9%
Smoking history	92.9%
Alcohol/drug use	100%
Dietary history	26.8%
Exercise habits	62.5%
Leg pain with exercise	23.2%

Table 2.

Laboratory Studies, Referrals, and Patient Education and Percent of Physicians Performing Recommended ADA Guidelines

Laboratory studies requested by physicians:	Percent of physicians who ordered each study:	Referrals and patient education:	Percent of physicians who performed the task:
Hemoglobin A1c	78%	Referred to ophthalmologist	39.3%
Lipid profile or cholesterol	86%	Referred to diabetes educator	26.8%
Microalbuminuria	41%
Metabolic profile	86%	Discussed monitoring glucose	55.4%
Thyroid-stimulating hormone	29%	Suggested change in diet	39.3%
		Suggested planned exercise	42.9%
		Told patient to stop smoking	66.1%
		Patient instructed in foot care	19.6%

DISCUSSION

This study demonstrates that unannounced standardized patients can gather useful data regarding physician compliance with published guidelines for care of persons with diabetes. SPs offer important advantages over assessments based on chart review and insurance claims^7,9,21–23. Chart abstraction is prone to false-negative results due to illegibility, lack of availability of the chart, or recording bias (physicians may not document every aspect of their exam)²⁴. Insurance claim data do not reflect lack of follow through by a patient on essential advice, such as obtaining a complete ophthalmologic evaluation. Claim data also lack sensitivity and do not capture the process elements or details of the physician—patient interaction that are an essential aspect of the visit²⁴. Unannounced SPs address these concerns by capturing the intent of the physician, thereby giving “credit” for all aspects of the encounter.

Though this study was limited by a small number of physicians, our results are consistent with recent reports from chart reviews using larger physician populations. We found that adherence to particular guidelines, such as foot examination, complete ophthalmologic examination, and metabolic control, is poor^25–27. Another limitation of our study is that physician performance was only assessed in the setting of an initial visit. Physicians may have been unfairly penalized for not performing interventions that they typically would accomplish at a follow-up visit as further data became available.

Accurate evaluation of physician performance is becoming even more critical as pay for performance measures are more widely adopted ²⁸. Many have expressed concern about the use of chart reviews and insurance claims as the only basis for evaluating performance, as they cannot account for documented patient factors, such as socio-economic and life circumstances, and limited health literacy that clearly impact patient outcomes. The ability to accurately capture additional aspects of the patient encounter, such using unannounced SPs, is an important consideration for policy makers seeking valid and reliable methods to evaluate physician performance.

In summary, we found that unannounced standardized patients can effectively evaluate physician practice behaviors and their adherence to practice guidelines in a manner not achieved by more traditional methods. This manuscript provides the details of this methodology so that others can adapt this approach within their own setting.